AI crawlers: which to allow, which to block, and how to do it

What are AI crawlers?

AI crawlers are the automated user agents that AI companies use to fetch web pages. They differ from Googlebot in what they do with the pages afterwards. OpenAI’s documentation draws the same three-way split that other vendors use: some bots collect content that may be used to train models, some build the index that powers AI search, and some visit a page only because a user asked a question.

• Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent. They collect content that may be used to train models, or in the case of Google-Extended and Applebot-Extended, they are control tokens that govern that use.
• AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index. They build the indexes that AI search products cite. Blocking them can remove you from that product’s answers.
• User fetch: ChatGPT-User, Claude-User, Perplexity-User. They fetch a page on request, for example when someone asks a question that needs it.

Which AI crawlers should you know?

The table lists every crawler Serpel checks, with what each vendor’s own documentation says. The purpose column is Serpel’s classification. CCBot, for example, counts as training because Common Crawl’s open data can be used by any organisation.

Should you block AI crawlers or allow them?

It depends on what you want from AI products. The groups are independent, so you can mix them. OpenAI states that a site can allow OAI-SearchBot to appear in search while disallowing GPTBot so its content is not used for training.

Two points often cause confusion. First, Google’s AI Overviews and AI Mode draw on the normal Search index, so they are controlled with Googlebot rules and snippet controls, not with Google-Extended. Google says robots.txt rules for Googlebot govern crawling for Search including AI features, and that nosnippet, data-nosnippet, max-snippet and noindex limit what is shown. Blocking Googlebot would remove you from Google Search, so never use it to opt out of AI. Second, a block affects future crawling. Anthropic describes a ClaudeBot block as excluding a site’s future materials from training datasets, so do not expect robots.txt to undo collection that already happened.

Ready-to-copy robots.txt examples

The files below are generated from the same crawler list the table uses. For a GPTBot robots.txt rule or a ClaudeBot robots.txt rule, add the crawler’s name as a User-agent line. Replace the example paths with your own and keep your existing rules for other crawlers. Because a crawler follows the group that names it and ignores the general group, the allowed bots repeat the Disallow line for the private path.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: MistralAI-Index
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Disallow: /admin/

User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Index
Disallow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /members/
Disallow: /downloads/

Several user-agent lines above one set of rules form a single group, which RFC 9309 allows. Product tokens are matched case-insensitively, so GPTBot and gptbot are the same rule. Search systems need time to notice a change: OpenAI says about 24 hours, and Meta says its crawlers may cache robots.txt for up to 24 hours.

Does robots.txt actually stop AI crawlers?

Only the ones that choose to obey it. RFC 9309 is explicit that the rules are not a form of access authorisation. Vendors document what their bots do:

• Training and search crawlers. Anthropic says its bots honour robots.txt, OpenAI manages OAI-SearchBot and GPTBot through it, and Meta presents it as the way to block Meta-ExternalAgent.
• User-triggered fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them.
• Verification. OpenAI, Perplexity, Anthropic, Common Crawl and Mistral publish IP ranges for their crawlers, so you can tell a real request from one that merely copies a name. Anthropic warns that blocking by IP address may not reliably guarantee an opt-out, so it recommends robots.txt.

If you need a hard block, use a firewall rule or authentication. The reverse problem is just as common: bot protection can block a crawler that robots.txt allows. OpenAI recommends allowing its published IP ranges for OAI-SearchBot, and Vercel’s analysis of AI crawlers mentions a firewall rule that blocks AI bots, which would also stop the ones you want.

How do you check which AI crawlers your site allows?

1. 1

   Test your robots.txt

   Paste your domain into the free robots.txt checker to see which AI crawlers your file allows and which it blocks.

2. 2

   Count the requests in your logs

   The command below counts requests per AI crawler in an access log. Google-Extended and Applebot-Extended never appear, because Google says Google-Extended has no separate user agent and Apple says Applebot-Extended does not crawl pages.

   grep -o -i -E "OAI-SearchBot|GPTBot|ChatGPT-User|Claude-SearchBot|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Meta-ExternalAgent|MistralAI-Index" access.log | sort | uniq -c | sort -rn

3. 3

   Audit it with every crawl

   Serpel’s site audit raises a warning when robots.txt blocks an AI search crawler, and serpel ai status lists which AI crawlers your robots.txt allows from the last crawl. AI visibility in Serpel also shows whether answers cite you.

Allowing the search crawlers is only the first step towards being cited. Our guide to SEO for ChatGPT covers the rest, and the llms.txt examples explain why llms.txt is a different file with a different job.