AI crawlers in robots.txt: search, user access and training

AI crawlers do three different things, and robots.txt handles each one under its own name. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot, which feeds Copilot) decide whether your site can appear in answers with a link. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because someone asked for it, right then. Training crawlers (GPTBot, ClaudeBot, CCBot) collect content that may go into model training; Google-Extended and Applebot-Extended are not crawlers but control names for the same purpose. You can block training and stay open to search, because the companies document each name with a separate function. Just don't expect robots.txt to stop anyone from accessing the site: it's a request, not a lock.

The information below comes from each company's official documentation, consulted on October 26, 2026. These pages change often; check the source before publishing new rules. If you're new to robots.txt, start with the guide to crawling and indexing.

How robots.txt picks the rules for each crawler

robots.txt is a text file at the root of the site (for example, https://www.example.com/robots.txt) with groups of rules. Each group starts with one or more User-agent lines carrying the crawler's name (the "token"), followed by Allow and Disallow lines with paths. The current standard is RFC 9309, and three of its rules explain most surprises:

1. The crawler looks for the group with its name, case-insensitively.

2. If it finds one, it obeys only that group. The User-agent: * group no longer applies to it.

3. Only when there is no group with its name does the crawler use the * group.

The second rule catches a lot of people: if you create a group for GPTBot and forget to repeat the Disallow: /cart/ that was in the * group, you've left the cart open to GPTBot. The same RFC says these rules "are not a form of access authorization" (RFC 9309, section 1).

The three kinds of AI-related access

Search and indexing crawlers

They crawl the site automatically to build the index the tool consults when answering. Blocking one of them reduces or removes the site's presence in that tool's answers; the exact effect depends on each company's documentation.

• OpenAI's OAI-SearchBot surfaces sites in ChatGPT's search features. According to OpenAI, sites that block this crawler don't appear in ChatGPT search answers, though they can still appear as navigational links. The company recommends allowing the crawler and the published IP ranges (OpenAI).

• Anthropic's Claude-SearchBot analyzes content to improve the relevance and accuracy of Claude's search answers. Blocking it prevents indexing and, according to Anthropic, may reduce the site's visibility in those answers (Anthropic).

• PerplexityBot surfaces and links sites in Perplexity results. The company says it is not used to collect content for foundation models (Perplexity).

• Microsoft's Bingbot is the Bing search crawler, and Copilot uses the same index. Bing's guidelines say Bing search and Copilot rely on the same crawling, indexing and ranking base, and list "blocking Bingbot in robots.txt" among the practices to avoid (Bing guidelines). Bing's official crawler list has no separate crawler for Copilot (Bing).

At Google, the crawler is Googlebot itself. Google's page on AI features in Search says that, since AI is part of Search, the robots.txt rules for Googlebot are the access control for those features (Google). Blocking Googlebot to get out of AI Overviews takes the site out of Search entirely. The section on Google further down shows the options that exist today.

User-triggered fetchers

A fetcher is a program that fetches one specific page, once, without going on to follow links. AI fetchers kick in when someone asks a question and the assistant decides to open a URL to answer it. Because a person made the request, some of the companies say robots.txt may not be followed in this case.

• OpenAI's ChatGPT-User is used for user actions in ChatGPT and in custom GPTs. OpenAI says it doesn't crawl the web automatically, that "robots.txt rules may not apply" because the action comes from the user, and that it doesn't decide whether content appears in search; that's OAI-SearchBot's job (OpenAI).

• Anthropic's Claude-User accesses sites when someone asks Claude a question. Here Anthropic takes a different path: it says its crawlers, including this one, respect robots.txt, and that blocking Claude-User prevents the system from fetching your content to answer the user (Anthropic).

• Perplexity-User visits pages to answer questions and may include the link in the answer. Perplexity says that, because a user requested the fetch, this fetcher "generally ignores robots.txt rules" (Perplexity).

Model training

These are the crawlers (or control names) that say whether content can go into AI model training.

• OpenAI's GPTBot crawls content that may be used to train OpenAI's foundation models. Blocking it indicates that the site's content should not be used in that training (OpenAI).

• Anthropic's ClaudeBot collects content that may contribute to model training. According to Anthropic, restricting ClaudeBot signals that the site's future materials should be kept out of training data (Anthropic). The word is "future": the page says nothing about content already collected.

• CCBot belongs to Common Crawl, a nonprofit foundation that maintains an open archive of web pages for anyone to download and analyze. The crawler reads robots.txt before fetching a page (Common Crawl FAQ). Common Crawl is not an AI company, but the archive is public and CCBot appears on training block lists; Cloudflare, for example, includes it among the crawlers blocked when a customer chooses to opt out of training (Cloudflare).

• Google-Extended is not a crawler. Google says this name has no user agent string of its own; crawling is done by the usual crawlers, and the token works only as a control in robots.txt. It determines whether content can be used to train future Gemini models and for grounding (giving the model content from the Search index at question time) in the Gemini apps and Vertex AI (Google).

• Apple's Applebot-Extended doesn't crawl anything either. Apple says it doesn't crawl web pages and only defines how data collected by Applebot may be used. Blocking it keeps the content out of training for Apple's foundation models, and the site can still appear in search results (Apple).

Reference table

How to block training and stay in search

OpenAI says each setting is independent: you can allow OAI-SearchBot to appear in search and block GPTBot to indicate that the content should not go into training (OpenAI). Google says Google-Extended doesn't affect a site's inclusion in Search and isn't used as a ranking signal (Google). Apple says pages that block Applebot-Extended can still appear in search results (Apple). Anthropic describes ClaudeBot and Claude-SearchBot as separate crawlers, each with its own effect.

The most common case looks like this:

# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including search crawlers
User-agent: *
Disallow: /cart/
Disallow: /my-account/

Sitemap: https://www.example.com/sitemap.xml

A group can have several User-agent lines in a row, and the rules apply to every name listed (RFC 9309). The search crawlers have no group of their own in this example, so they follow the * group.

If you want to make explicit who is allowed, create a group for the search crawlers and repeat the * group's restrictions in it, because of the specific-group rule covered at the start:

# Search in AI tools: allowed, except private areas
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/
Disallow: /my-account/

# Training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /my-account/

Sitemap: https://www.example.com/sitemap.xml

You can also block training on only part of the site. Google itself uses an example like this with Google-Extended:

User-agent: Google-Extended
Allow: /archive/1Q84
Disallow: /archive/

Two things to watch. Applebot follows Googlebot's rules when robots.txt has no group for it (Apple); a group that blocks Googlebot ends up blocking Apple's search too. And Bing has no separate name for training or for Copilot in robots.txt: blocking Bingbot takes the site out of Bing and Copilot at the same time.

Bing and Copilot: the control is in the meta tags

To limit use in Copilot without leaving Bing, Microsoft uses the robots meta tag (or the X-Robots-Tag header), page by page (Bing):

• With noarchive, the page isn't linked in Chat and Copilot, and the content isn't used to train Microsoft's foundation models.

• With nocache, Copilot shows only the URL, title and snippet, and only those fields may go into training. With both tags, nocache applies.

• With noindex, the page leaves Bing's index and isn't used for training either.

Changing name="robots" to name="bingbot" makes the tag apply only to Bing. Bing's current guidelines warn that nocache reduces the depth of citations in Copilot (Bing).

Google: AI Overviews, AI Mode and Gemini are different controls

Google has three controls, and each one acts on something different:

1. Google-Extended, in robots.txt, controls training of future Gemini models and grounding in the Gemini apps and Vertex AI. According to the Search Console help, it is also the way to limit training of the models that generate answers in Search's generative AI features (Search Console). It doesn't affect presence or ranking in Search.

2. The Search Console generative AI control, available to all sites since August 31, 2026, lets you exclude the site's links and content from AI Overviews, AI Mode and Discover's generative AI features. Google says this control is not a ranking signal for the rest of Search and doesn't affect AI training (Search Console).

3. The on-page snippet controls (nosnippet, data-nosnippet, max-snippet and noindex) limit what appears from the page in Search, including the AI features (Google).

What robots.txt doesn't solve

It isn't access control

robots.txt expresses a preference. RFC 9309 says the rules are not access authorization, and Google says the file is not a way to keep a page out of Google: for that, use noindex or protect the page with a password (Google). Cloudflare says the same: following robots.txt is voluntary, and some crawler operators ignore the directives (Cloudflare). Content that third parties must not read needs a login, not a Disallow.

User-triggered fetchers are the clearest case. If you need to stop ChatGPT-User or Perplexity-User, robots.txt may not be enough, according to the companies themselves. The block has to happen on the server or in the web application firewall (WAF), by user agent checked against the published IP ranges.

The user agent can be spoofed

Any program can present itself as GPTBot or Googlebot. Bing says user agent strings are easy to spoof (Bing), and Common Crawl says it knows of crawlers posing as CCBot (Common Crawl). Before drawing conclusions from logs or allowing a crawler in the firewall, confirm where it comes from:

The lists change. Bing asks you to refresh the IP list daily, and Perplexity recommends automating updates to WAF rules. Anthropic warns in the opposite direction: blocking its IPs instead of using robots.txt may not guarantee exclusion, because it stops the crawler from reading the file with your preference.

The CDN or WAF may block before robots.txt

If the site sits behind a CDN or WAF with bot management, the request can be stopped before it reaches the server, whatever robots.txt says. At Cloudflare, for example, the AI crawler policies have separate options for search, agent and training, each with blocking on all pages, blocking only on pages with ads, or allowing. For new domains, since September 15, 2026 the default blocks training and agent crawlers on pages with ads and keeps search allowed (Cloudflare). Cloudflare can also insert its own rules at the top of the robots.txt it serves when preference syncing is turned on (Cloudflare).

In practice, the robots.txt you edited may not be the one crawlers receive, and a crawler allowed in the file may be blocked at the edge. Open the public robots.txt URL to see the content actually served, and check the CDN's firewall events. Perplexity, for example, advises allowing its crawlers in the WAF when the site uses one.

So, should I block GPTBot, ClaudeBot and the like?

This part is a recommendation, not documented fact. It's a business decision, and robots.txt lets you make the two choices separately:

• If the site depends on being found (a store, a service, content that brings in customers), keep the search crawlers allowed. Blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot or Bingbot reduces or removes the site's presence in those tools' answers.

• Blocking training is a choice about how your content is used. According to OpenAI, Google and Apple documentation, it doesn't affect those companies' search. None of them promises any effect on content already collected.

• User-triggered fetchers bring visits from people who asked an assistant something. Blocking them makes sense for areas that shouldn't be read by automation; elsewhere, the block tends to just take the site out of those answers.

This split doesn't apply to sites that want no AI use at all. In that case, the way out is to block every name and accept the loss of visibility in these tools; on Google and Bing, that means using the page-level controls and Search Console, because blocking Googlebot or Bingbot takes the site out of traditional search too.

A point that often comes up alongside this: the llms.txt file doesn't control any of it. It's a proposed site summary for AI agents, with no permission effect. The details are in the article on llms.txt.

How to check your site

WebVyze's robots.txt validator is free and checks the file's syntax, the declared sitemap and which crawlers may crawl a path (Google, Bing, ChatGPT, Perplexity, Claude and Gemini), with search and training kept apart. It reads the robots.txt the site serves publicly; blocks set up in the CDN or firewall have to be checked in those services' dashboards.