HTTPS surface reachable (robots ✓, sitemap ✗, title ✓)
Why it matters: Public files — robots.txt, sitemap.xml, head meta — are what attackers see first during reconnaissance. Misadvertised paths, stale sitemaps, and verbose generators leak more than intended (ISO 27001 A.8.9).
robots.txt
present
User-agent: *
Disallow: /info/p.gif
Disallow: /p/
Disallow: /r/
Disallow: /bin/
Disallow: /caas/
Disallow: /blank.html
Disallow: /includes/
Disallow: /_td_api
Disallow: /tdv2_fp
Disallow: /nel_ms
Disallow: /fp_ms
Disallow: /sports_fp_ms
Disallow: /search_ms
Disallow: /_tdpp_api
Disallow: /_remote
Disallow: /_multiremote
Disallow: /_tdhl_api
Disallow: /digest
Disallow: /fpjs
Disallow: /myjs
Disallow: /news/m/
User-agent: ADmantX
User-agent: AlphaBot
User-agent: anthropic-ai
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
User-agent: BLEXBot
User-agent: Buzzbot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: claritybot
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: FacebookBot
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GPTBot
User-agent: huggingface
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: magpie-crawler
User-agent: Meltwater
User-agent: Neevabot
User-agent: news-please
User-agent: NewsNow
User-agent: Nutch
User-agent: omgili
User-agent: omgilibot
User-agent: panscient.com
User-agent: Perplexity-ai
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PiplBot
User-agent: scoop.it
User-agent: Scrapy
User-agent: Seekr
User-agent: SentiBot
User-agent: SeznamBot
User-agent: TurnitinBot
User-agent: YouBot
User-agent: ZumBot
Disallow: /
User-agent: Claude-SearchBot
User-agent: OAI-SearchBot
Disallow: */articles/
Sitemap: https://www.yahoo.com/news/weather/sitemap.xml
Sitemap: https://www.yahoo.com/news-sitemap-index.xml
Sitemap: https://www.yahoo.com/sitemap-index.xml
head
- title
- Yahoo | Mail, Weather, Search, Politics, News, Finance, Sports & Videos
- description
- Latest news coverage, email, free stock quotes, live scores and video are just the beginning. Discover more every day at Yahoo!
social
no OpenGraph or Twitter meta tags found