ai-crawler checker
check whether a domain's robots.txt has told the major AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot — anything at all. part of the domainposture.com domain dossier.
ai-crawlers
infochecking ai-crawler policy…
Want every check for e2b.dev on one page? Open the full dossier →
Overview
robots.txt has governed search-crawler behaviour since 1994, but AI companies reused it as the de-facto opt-out mechanism for training and answer-engine crawlers. OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended, Perplexity's PerplexityBot, Common Crawl's CCBot, and Meta's meta-externalagent each respect a User-agent group matching their name. the catch: if a site has no robots.txt at all, or has one that never mentions these agents by name, every one of them defaults to fully allowed — a fallback `Disallow: /` under `User-agent: *` does NOT bind AI crawlers unless they're also matched by an explicit or wildcard group that includes rules. this tool fetches robots.txt, groups the User-agent/Disallow/Allow directives the same way a crawler would, and reports each of the six major AI agents as allowed, blocked, or unspecified. it does not fetch or reproduce site content — it reports the declared policy only. this is a governance stance, not a vulnerability: many sites deliberately allow AI crawlers for visibility in answer engines, and many haven't made a decision either way. an audit records which case it is.
How to use
- enter a bare domain — public fqdn only. no scheme, no port.
- read the robots.txt stance — the tool fetches https://<domain>/robots.txt and groups its directives the way a crawler parses them — a User-agent line starts a group, Disallow/Allow lines belong to the group above.
- check each agent's stance — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and meta-externalagent are each reported allowed, blocked, or unspecified. unspecified agents fall through to whatever the site's default is — often nothing, meaning allowed.
Examples
input
site with `User-agent: GPTBot\nDisallow: /`output
GPTBot: blocked · ClaudeBot: unspecified · Google-Extended: unspecified · …input
site with no robots.txtoutput
hasRobots: false · all six agents: unspecifiedCommon mistakes
a wildcard block doesn't cover named AI agents the way you'd expect— `User-agent: *` followed by `Disallow: /` is the group crawlers fall back to when no more specific group names them. but the moment a site adds even one named group (e.g. `User-agent: GPTBot`) without a Disallow rule, that agent is treated as explicitly allowed, not caught by the wildcard block — the specific group wins.declaring a stance and enforcing it are different things— robots.txt is an honour-system convention, not a technical control. compliant crawlers (the major AI labs generally are, for now) respect it. it does not stop a scraper that ignores robots.txt entirely — that requires rate limiting, IP blocking, or a WAF rule.blocking training crawlers can also block answer-engine retrieval— some of these agents (e.g. Google-Extended, PerplexityBot) are used both for model training and for live retrieval when a user asks a question in an AI answer engine. blocking them opts out of both — there's currently no widely supported way to allow retrieval while blocking training for every vendor.the policy can live off-domain and still count, or not count at all— if requesting /robots.txt redirects to a different domain, this check treats the response as absent rather than crediting a policy the domain itself didn't publish — the same rule crawlers should apply.
FAQ
what happens if I don't declare an AI-crawler policy at all?
every major AI crawler defaults to allowed. GPTBot, ClaudeBot, Google-Extended, and PerplexityBot will crawl the site for training data and answer-engine retrieval unless robots.txt tells them otherwise.
does blocking these crawlers hurt my SEO?
no — these are separate from the search-indexing crawlers (Googlebot, Bingbot). blocking GPTBot doesn't affect Google Search or Bing ranking; it only opts the site out of AI training and, for some agents, AI answer-engine retrieval.
is there a standard list of AI crawler user-agents?
not an official one — each vendor documents its own agent string (OpenAI, Anthropic, Google, Perplexity, Common Crawl, and Meta all publish theirs separately). this tool checks the six most commonly cited in AI-crawler discussions: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and meta-externalagent.
should I block or allow AI crawlers?
it's a business decision, not a security one. allowing them can increase visibility in AI answer engines; blocking them opts content out of training corpora. an audit records whichever choice was made — or that none was made.
can ai agents call this?
yes — dossier_ai_crawlers on the mcp endpoint at domainposture.com/mcp/mcp.
how do I block a specific AI crawler?
add a group to robots.txt naming the agent exactly, followed by a Disallow rule, e.g. `User-agent: GPTBot` then `Disallow: /`. the agent name must match what the crawler sends — check the vendor's published documentation for the exact string.
Related tools
web surface inspector— fetch robots.txt, sitemap.xml, and the home page's <head> to summarise a domain's public-web surface.llms.txt checker— check whether a domain publishes an llms.txt content index for AI agents.security.txt checker— check whether a domain publishes a security.txt (RFC 9116) so researchers have a way to report vulnerabilities.
References
Free beta
Unlock your full scan (free beta)
Add one DNS TXT record to prove you own the domain, then we run a deeper active scan — exposed secrets and config — and send you a private report. Free beta, no payment.
- Finds exposed .env / .git / backups and other sensitive files
- Verified by a DNS TXT record, so only you can run it on your domain
- A private report emailed straight to you