Blog
Who Is Your robots.txt Blocking? AI Crawlers and the Cost of Being Invisible
AI search visibility strategy starts with AI crawler robots.txt
Most crawlers have a purpose and a name, so you can block or allow each crawler separately in your robots.txt file. For instance, some AI crawlers collect data for AI training purposes. These include:
- GPTBot, OpenAI's crawler for training data
- ClaudeBot, Anthropic's crawler for training data
- CCBot, Common Crawl's crawler for training data
Other AI crawlers exist for AI search engines. These crawlers collect data to serve up as AI search results. These crawlers include:
- OAI-SearchBot, OpenAI's crawler for its AI-powered search engine
- Claude-SearchBot, Anthropic's crawler for its AI-powered search engine
- PerplexityBot, Perplexity's crawler for its AI-powered search engine
Some AI crawlers collect data on behalf of a user when that user submits a query in an AI tool. For instance, if a user submits a query in ChatGPT, Claude, or Perplexity that can only be answered by collecting information from your site, these user-based crawlers will collect that data:
- ChatGPT-User, OpenAI's crawler for user queries
- Claude-User, Anthropic's crawler for user queries
- Perplexity-User, Perplexity's crawler for user queries
Google and Apple have robots.txt instructions related to AI crawlers, but these companies don't have crawlers by these names. According to Google's common crawlers documentation, Google-Extended doesn't have its own user agent. This robots.txt instruction controls whether Google can use content crawled by Googlebot for training and grounding in the Gemini app. Similarly, according to Apple, Applebot-Extended does not crawl webpages. Instead, Applebot-Extended tells Apple whether Applebot's data may train Apple's models.
Block AI training crawlers (but not others)
Based on this list of AI crawlers, you might think the answer is obvious: block all the AI crawlers that you don't want indexing your site. Don't do it.
Effective September 15, 2026, if you use Cloudflare to serve your site, choosing either Block or Block on pages with ads in Cloudflare's AI traffic blocking controls blocks Googlebot, Bingbot, and Applebot at the CDN level (search included, in Cloudflare's words). This will remove your site from Google. (And Bing and Apple, but let's stick with Google for now.)
My suggestion: if you're opposed to your content being used for AI training, block the crawlers that collect training data (e.g., GPTBot and ClaudeBot). Allow access to the AI crawlers that collect data for search engines (e.g., OAI-SearchBot and Claude-SearchBot). Decide which crawlers to allow access based on the AI platforms where you want to show up in responses to customer queries. And be sure that any blocking you're doing via robots.txt isn't being trumped by blocking at the CDN level.
For example, if you want to opt out of AI training while preserving visibility in AI-powered search engines, your robots.txt might include these rules:
# Training opted out, search bots left open. Bot list checked October 2026.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /The Google-Extended entry would also opt your site out of grounding in the Gemini app. If you want your site included in grounding for Gemini, don't include this entry.
Here's why it's important to allow access for crawlers that collect data for AI search engines. According to OpenAI's publisher FAQ, if you want your content to appear in ChatGPT via snippets and page summaries, you shouldn't block access to OAI-SearchBot in your robots.txt file. And if you opt out of OAI-SearchBot, OpenAI cautions that this means your content will not be shown in ChatGPT search answers, though can still appear as navigational links.
Similarly, in an article last updated April 7, 2026, Anthropic explains that if you disable Claude-SearchBot in robots.txt, this may reduce your site's visibility and accuracy in user search results. And blocking access to Claude-User may reduce your visibility for user-directed web search.
Is it worth blocking GPTBot, ClaudeBot, and other crawlers?
Some sites have good reasons to block access to crawlers that collect data for AI training. For instance, according to Cloudflare's analysis of crawl to refer ratios, in July 2025, Anthropic crawled approximately 38,000 pages for every referral it sent Cloudflare customers, compared to approximately 1,100 pages for OpenAI and around 5 pages for Google.
And according to a 2026 study by Zyte that examined robots.txt files for 11,100 landing pages, 64% of sites belonging to newspapers include at least one Disallow rule for AI crawlers.
But don't forget that your goal is to get found, including in AI platforms. That's particularly true for service pages and help center pages. It's your call whether to opt out of AI training, but make sure you don't opt out of OAI-SearchBot, Claude-SearchBot, or PerplexityBot by mistake.
If you want to block access to members-only or licensed research pages, you can block those with Disallow rules in robots.txt. Just know that you may want to implement that blocking at the CDN level rather than robots.txt for enforcement purposes.
Can you opt out of AI Overviews and AI Mode in Google Search?
Yes. But not using robots.txt.
Google explains that blocking access to the Google-Extended token doesn't keep your site out of AI Overviews or AI Mode. Google says that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal.
Remember that Google uses regular crawls with Googlebot to build indexes for its AI features in Search. And Google notes that directives in robots.txt for Googlebot apply to all Google Search features.
Instead, you can opt out of AI Overviews and AI Mode in Google Search via a setting in Search Console.
Google notes that, as of August 31, 2026, the opt-out setting for its generative AI features is available to all websites worldwide, with inclusion as the default option. And Google warns that if you choose to exclude your website, please note that you won't receive any traffic or impressions from these features.
Note that this setting is separate from opting out of AI training with Google-Extended. And don't forget that AI Overviews can impact CTR to classic results, including ads, as we explored in our analysis of AI Overviews' impact on CTR for paid search results.
So consider whether giving up AI Overviews and AI Mode for all queries aligns with the pages you most want to rank for.
Cloudflare AI training crawler defaults
If you use Cloudflare, you may want to check your settings to see whether you're blocking AI crawlers. On July 1, 2025, Cloudflare changed its default setting for new domains to block AI training crawlers unless a site owner specifically opted out, as part of its Content Independence Day.
Beginning on September 15, 2026, the AI training crawler default setting will be as follows:
- New domains that don't use ads: Allow
- New domains that use ads: Disallow AI Training
- Existing domains using the legacy Block AI crawler setting: Disallow AI Training
In a Sept. 15 post, Cloudflare also cautions site owners that choosing Block will stop Applebot, Bingbot, and Googlebot from reaching your site, including on pages that include ads. If you're using the Block setting, you may want to consider whether this is what you intend. (You may want to check to see if search engines can still crawl your site if you currently have either of these settings.)
Fewer than 1% of Cloudflare sites block search engine crawlers, according to the post, but 17% use some method to block AI training bots.
Check if your CDN is blocking AI bots If you use a CDN or web application firewall, it might block AI crawlers before they even get to your robots.txt file, so your robots.txt file allowing AI crawlers isn't enough by itself to make sure AI crawlers can access your site. In its crawler documentation, Perplexity notes that if you're using a web application firewall, you may need to allowlist Perplexity's bots.
How to check whether AI crawlers are being blocked
Export 30 days of log data from your server or CDN.
Look for requests from AI search bots, including OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot.
See if they're getting 403 errors or other signs that your site isn't allowing them (such as a challenge page).
If you're using Cloudflare, you may be able to check your AI crawler setting and compare it to analytics within your Cloudflare dashboard to see if AI crawlers are getting blocked.
Your organization's IT or security team may control your CDN settings, and you may not know whether it's blocking AI bots.
Also consider checking whether wildcard rules in your robots.txt file (such as ones that disallow all bots) are unintentionally blocking AI crawlers.
AI crawlers can ignore robots.txt
Crawlers associated with user-triggered queries may ignore your robots.txt rules, according to OpenAI, Perplexity and Google. For example, OpenAI notes that robots.txt rules may not apply to ChatGPT-User and Perplexity notes that Perplexity-User generally ignores robots.txt rules. Google notes that user triggered fetchers generally ignore robots.txt rules.
As a result, blocking AI crawlers in robots.txt may not prevent AI systems from including content from your site in AI-generated answers.
News sites that block AI crawlers in robots.txt saw website visits drop by 7%
Hangcheng Zhao and Ron Berman at Rutgers and Wharton estimate that blocking AI crawlers in robots.txt is associated with a ~7% drop in weekly website visits within six weeks in a working paper (last revision April 2026).
The decline was evident using SimilarWeb and Semrush data. The authors also saw a similar decrease using Comscore browsing panel data (though it was not statistically significant). The data spans November 2022 to May 2024. Their primary hypothesis for the cause of the decrease is that blocking AI crawlers leads to less brand exposure via AI systems.
70% of top news sites that block ChatGPT-User in robots.txt still appear in AI citations
In March 2026, BuzzStream found that 70.6% of top news sites that disallow ChatGPT-User in robots.txt and 82.4% of top news sites that disallow OAI-SearchBot in robots.txt still appeared in citations in a sample of 4 million AI-generated citations.
These studies only looked at news sites (Zhao and Berman look at blocking AI bots via robots.txt, in general). So don't read too much into these studies for your brand. But the fact that some blocked websites are still being cited suggests that the protection afforded by blocking AI crawlers may be limited. That's another reason to consider blocking on a per-bot basis, rather than across the board, and to measure the impact for your site using your own referral data.
1 in 4 websites disallow all bots in robots.txt
In its State of Web Access 2026 report, Zyte found that 12.7% of sampled landing pages disallow AI bots using a Disallow directive in robots.txt and another 27.2% disallow all bots, meaning that AI bots are also blocked by default, not by intent.
Which queries should you never block AI crawlers from?
Decisions about which AI crawlers to block or allow should be owned by a single person or team and should be made deliberately rather than defaulting to whatever settings your CDN or a plugin recommends. The owner of AI crawler access should be the same person accountable for search traffic.
AI crawler policy should be documented as part of your broader generative engine optimization strategy, with specific rules for each bot and use case (training, indexing, user fetches) and for each path on your website. Review it quarterly against logs and AI crawlers referenced in your robots.txt, and whenever a major operator adds or renames a bot (e.g., Cloudflare reports that Microsoft's robots.txt training preference is targeted for early 2027).
Start with buyer questions. Pull queries from the queries report in Search Console and the paid search terms report, add questions your sales team hears on calls, and pay special attention to long tail questions. Analyze these at the query level and note the page you want to rank for each query. You never want to block AI crawlers from the pages that answer these queries.
Test if those pages are visible in AI answers. Paste each of these questions into ChatGPT, Perplexity, Gemini and Google's AI Mode, and note if your pages are cited. Do this more than once, as the answer can change each time you ask the same question. Our review of AI visibility tools explains why the scores can be so different from one tool to another. Check that the pages receive a status code of 200 when requested by the relevant search bot. Monitor retrieval bot traffic and AI referral traffic (ChatGPT appends utm_source=chatgpt.com to links) for several weeks after making any changes.
Should you use llms.txt?
llms.txt doesn't allow or block crawlers. It's a proposed plain text file that summarizes a site for language models. The proposal notes that robots.txt and llms.txt have different purposes and the author expected llms.txt would have more of an impact for agents at inference time than for training. Google's AI optimization guide says Google Search ignores llms.txt, and publishing one neither helps nor harms visibility in Google Search.
One study by Search Atlas analyzed llms.txt files on 13,911 domains and found that having an llms.txt file did not predict if a domain was cited by an LLM when domain authority was included as a variable (note that Search Atlas also sells a Search Atlas AI Visibility tool, so consider this vendor research; last updated in June 2026). Access is controlled at the edge and in robots.txt, so it makes sense for the same owner to make these decisions in one place.
