Should You Block AI Crawlers From Your Website?

Should you block AI crawlers? Most businesses should leave search and user bots open and decide on training bots deliberately. What 20 big sites do, and where blocks leak.

RankControl10 min read
Should You Block AI Crawlers From Your Website?

On September 26 we pulled the robots.txt files from 20 well-known websites. Every one of the five news sites had rules that named AI crawlers. Of the other 15, only three mentioned an AI crawler by name at all. Two more sites, Vercel and Twilio, carried a line you've probably never seen: Content-Signal: search=yes, ai-input=yes, ai-train=no.

The file that stuck with me was HubSpot's. It's 12.6 KB long, with over 350 Disallow lines, and there's no AI bot anywhere in it.

So, do you block AI crawlers? For a business that sells something and wants to be found, mostly no. Leave AI search bots and user-triggered fetchers free to visit, and decide about training bots as a separate matter. Below I cover why, what the 20 sites actually do, and why a block often leaks even when you mean it.

AI crawlers come in three kinds

People say "AI crawler" as if it were one thing, but it covers three jobs, and each big vendor now runs a separate user agent per job:

KindWhat it doesExamplesBlock it and you lose
TrainingCollects pages that may train future modelsGPTBot, ClaudeBot, Meta-ExternalAgent, CCBotNothing visible today
SearchBuilds the index AI answers draw fromOAI-SearchBot, Claude-SearchBot, PerplexityBotYour place in that engine's answers
User fetchOpens a page because someone asked about itChatGPT-User, Claude-User, Perplexity-UserLittle, since most of these ignore the block

OpenAI's developer documentation page Overview of OpenAI Crawlers, explaining that OAI-SearchBot and GPTBot settings are independent

OpenAI lays the split out on its crawler page. You can let OAI-SearchBot in for search results while GPTBot stays disallowed. The page is blunt about the reverse case: "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Anthropic's three bots are split the same way, and Anthropic cautions that shutting out Claude-User could hurt how visible you are in user-directed search.

Two entries you'll find on most crawler lists don't crawl anything. Google-Extended is really a control token for Gemini training and grounding, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Applebot-Extended, Apple's version, "does not crawl webpages," so a block only keeps Apple from training on your pages.

For the full set of user agent strings, see our complete AI crawler list.

What blocking each kind costs you

Start with training bots, because a block there costs you very little that you'd notice. Your content just stops going into that vendor's future models, and your analytics look the same tomorrow as today. Over the long run there may be a cost you can't measure well: a model trained without your pages might have a thinner picture of your product when it answers from memory and not from a live search.

Search bots are the expensive ones to block. OpenAI's docs are unambiguous: block OAI-SearchBot and ChatGPT search answers won't include you. If you run a SaaS company, picture a buyer asking for the "best help desk for a 20-person team" and your product missing from the answer.

Blocking the user fetchers is mostly futile, and the vendors say as much. Per OpenAI, robots.txt rules "may not apply" to ChatGPT-User, because a person started the visit. Perplexity's docs go further and say Perplexity-User "generally ignores robots.txt rules." And Meta-ExternalFetcher, by Meta's own account, "may bypass robots.txt." With three vendors putting the same caveat in writing, I'd read robots.txt as a polite request that these fetchers can refuse.

What 20 big sites actually block

Our September 26 census, sorted by kind of site, looks like this. A site gets a yes under "Name an AI bot" when its robots.txt names at least one AI crawler in a rule, beyond the generic User-agent: * rules everyone has. Content-Signal is the newer preference line I'll get to further down.

GroupSites checkedName an AI botUse Content-SignalWhat stood out
NewsNYT, WSJ, Guardian, Reuters, Washington Post5 of 50NYT blocks every AI bot we checked
SaaSHubSpot, Salesforce, Zendesk, Atlassian, Intercom0 of 51Intercom's signal says yes to all three uses
EcommerceNike, Target, Best Buy, Wayfair, Chewy1 of 50Best Buy files AI bots under "AI search and discovery bot access"
Developer toolsStripe, GitHub, Vercel, Cloudflare, Twilio2 of 53Vercel and Twilio say no to training, yes to search

GitHub has the most careful setup of the 20. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot each get a one-second crawl delay there. They can reach marketing pages such as /pricing and /features, but repository pages like commit histories and forks are off limits.

The news sites line up with a bigger study. In July, FT Strategies reviewed 70 large publishers. It found 39 of them fully blocking GPTBot, while just 16 blocked OAI-SearchBot. The authors see publishers shutting out training crawlers while taking "a more selective approach to those that feed AI-generated answers." They also admit the sample isn't representative, since the publishers were picked for being active in AI licensing debates.

One pattern in our census deserves a warning. At three of the five news sites, OpenAI's search and user bots got more access than their Anthropic or Perplexity counterparts. That probably reflects each publisher's business relationships more than any view on which bots behave, so don't copy a news site's robots.txt for your software company.

Your competitors are getting cited by AI. You're not.

Every day without citation tracking is a day your competitors pull ahead in ChatGPT, Perplexity, and Claude.

Show me who's getting cited →2-minute overview · real case-study numbers

The leaky block

We also tested whether blocks hold up. Six of the 20 sites each got one question, asked both of GPT-5.5 with web search and of Perplexity's API. NYT, the Washington Post and Reuters block Perplexity's crawler, while Stripe, Zendesk and Nike block nothing.

The open sites were cited from their own pages, as you'd expect. Stripe's pricing page turned up, and so did Nike's returns policy and Zendesk's pricing page, although ChatGPT pointed to a Zendesk comparison page and not the pricing one.

Most of the blocked sites were cited too. PerplexityBot is blocked on the Washington Post's main domain, yet both engines cited pages from its help center and subscription subdomains, and those run their own robots.txt files. Reuters answers came from reutersagency.com, a different domain, plus one reuters.com article Perplexity cited. Perplexity went straight to nytimes.com subscription and help pages, despite the Times blocking every AI bot we looked at.

The name I've given this is the leaky block. The rule has a gap, whether that's a subdomain, a sibling domain or a user-triggered fetcher, and the content finds another way into AI answers through it. ChatGPT on the Times was the one block that held, and it cited a help portal run by a third party instead of nytimes.com.

Two caveats before you lean on any of this. We asked one question per site, so the check proves leaks happen and says nothing about how often. We also can't see how Perplexity reached the Times pages. It could have been its own crawler, a user fetch or an older copy. Cloudflare accused Perplexity in August 2025 of using a generic Chrome user agent to crawl sites that had blocked it, at three to six million requests a day, which Perplexity denied.

If blocking is your call, whether blocking stops Perplexity from citing you covers the enforcement layers.

Robots.txt is a request, and so is what replaces it

Cloudflare's Content Signals Policy, from September 2025, gives you a middle option. You add a Content-Signal line to robots.txt and answer three questions on their own: search indexing, live input for AI answers, and training. Both Vercel and Twilio set ai-train=no and leave the other two open. I suspect that's what most software companies want, and Disallow rules alone can't say it.

Cloudflare's blog post from September 24, 2025, Giving users choice with Cloudflare's new Content Signals Policy, by Will Allen

It's still only a preference. No crawler is obliged to honor the line, and the IETF group working on a standard for AI preferences puts technical enforcement outside its scope. In September, Search Engine Journal boiled down Helen Pollitt's advice: block at the highest layer you can reach, meaning a firewall or CDN, since a robots.txt rule only stops bots that decide to obey it.

Block AI crawlers in robots.txt or at the server level? Helen Pollitt's answer: Go as high up the stack as you can. WAF if you have one, CDN if you don't, server as a last resort. Robots.txt only works on bots that choose to respect it. More in this week's Ask An SEO 👇 https://t.co/XvVUAX7W13

SearchEngineJournal®@sejournalSep 3, 2026

Blocking at the firewall comes with its own trap. One r/TechSEO poster turned on Cloudflare's AI training block in August and then saw Googlebot and Bingbot run into 403 errors on the sitemap. A commenter replied that Cloudflare documents this, since those bots count as multi-use crawlers there, and Google's John Mueller chimed in with an offer to look into it.

r/TechSEO· u/TheChiuaua· Aug 3, 2026

Has anyone else run into Cloudflare AI Crawlers blocking Googlebot/Bingbot?

I'm testing Cloudflare's new AI Crawlers & Scrapers feature and noticed something unexpected. When I set AI Training = Block, both Googlebot and Bingbot start receiving HTTP 403 responses when trying to fetch my sitemap. As soon as I disabl...

↑ 9 upvotes24 comments
Via Reddit

The default we'd pick for a business

If your company wants to be found, this is the robots.txt I'd start from:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Four training crawlers are blocked, and all the search bots and user fetchers stay open. Maybe you'd rather models learned from your pages, and plenty of software companies would. In that case, remove the four training blocks and flip ai-train to yes. Legal and marketing should decide that together, and either answer is fine provided the search bots remain open.

Once it's live, make sure it held. Go through the firewall or CDN settings for any AI switch that overrides robots.txt, and look at each subdomain that has its own file. The setup is in our robots.txt and llms.txt checklist, and a server log audit tells you which bots showed up after the change.

Plan on an hour for writing the file and going over the firewall settings. After that, about 30 minutes a month is enough to check that the bots you allowed keep visiting.

RANKCONTROL

200+ SaaS teams already track their AI citations.

They know exactly when ChatGPT mentions their brand, and when it stops. Do you?

Show me the plan →One plan · everything included

When blocking makes sense

The clearest case is a publisher whose business is the article itself. When an AI answer assembled from your reporting takes the place of a visit, blocking is a reasonable stance, and it strengthens your hand in licensing talks.

Over on r/webdev the complaint is different. In July a developer posted that a page-speed article of theirs had been scraped and put to use elsewhere within ten days, and the thread passed 200 upvotes. The useful advice in the replies was filing a DMCA request with Google over the copied URL, which, according to one commenter, usually gets scraped pages taken down in a couple of weeks. No robots.txt rule would have stopped that scraper, which never asked for permission.

r/webdev· u/svvnguy· Jul 29, 2026

Web scraping and content stealing is out of control

10 days. That's all it took for one of my articles on page speed to get scraped and reused in the marketing SEO slop of some online company, and then get found by me (purely by accident). I know it's my content because I think and write eve...

↑ 226 upvotes81 comments
Via Reddit

For everyone else, the bigger danger is a block you never meant to set, such as a CDN switch, a template you inherited or a newspaper's file you copied, that ends up locking out the search bots you need.

Watching what the bots do

You can do the upkeep by hand. Go back over robots.txt and the firewall whenever your CDN changes, read through your logs each month, and put your buyer questions to ChatGPT and Perplexity to make sure you're still being cited.

Or you can let RankControl do the watching. Our analytics track both AI crawler visits and AI-referred visits, which shows you the bots that still arrive after a change. The AI visibility tracking runs up to 50 buyer questions each week on seven AI engines. Start with the free AI visibility check to see where you stand today.

RANKCONTROL

How often does ChatGPT mention your brand?

Most founders have no idea. The answer might surprise you.

Show me my mentions →50 queries tracked · all 7 AI models

Frequently Asked Questions

If you want customers to find you, I'd leave AI search bots and the fetchers users trigger alone, since blocking them takes you out of AI answers. Training bots are their own question. You can block those on legal or licensing grounds and your search visibility won't suffer.

It doesn't, because GPTBot only collects training data for OpenAI. ChatGPT's search answers depend on OAI-SearchBot, and OpenAI says opting out of that bot keeps a site out of ChatGPT search answers. You're free to block GPTBot and still allow OAI-SearchBot.

No. Google says Google-Extended has no effect on whether your site is included in Google Search, and it isn't a ranking signal. All it decides is whether Gemini can train on your content and use it for grounding.

The big vendors say their training and search crawlers do. User-triggered fetchers are another matter, though. OpenAI warns that robots.txt rules may not apply to ChatGPT-User, and by Perplexity's own account, Perplexity-User generally ignores robots.txt.

Most of them block training and let search in. FT Strategies looked at 70 large publishers and found 39 of them blocking OpenAI's training crawler GPTBot outright, while only 16 blocked OAI-SearchBot.

RANKCONTROL

Your competitors are already optimizing for AI search

Content that ranks on Google and gets cited by AI search engines. Published on your domain. Citations tracked weekly.

Related Articles

What Are AI Bot Visits And Should Marketers Track Them?

Nov 12, 2026 · 8 min read

What Are AI Bot Visits And Should Marketers Track Them?

AI bot visits are requests from AI crawlers, fetchers and agents. The four kinds, why GA4 never shows them, what they tell you, and how much tracking is worth it.

Cloudflare Bot Fight Mode And SEO: What To Check Before Blocking Crawlers

Nov 11, 2026 · 9 min read

Cloudflare Bot Fight Mode And SEO: What To Check Before Blocking Crawlers

Does Cloudflare Bot Fight Mode hurt SEO? What it blocks, why you can't make exceptions on the free plan, the September 15 AI crawler change, and a checklist to run first.

Does Your Domain Name Help You Get Cited By AI Search?

Nov 6, 2026 · 8 min read

Does Your Domain Name Help You Get Cited By AI Search?

A $224 test checked 7,374 keywords, 3,048 new domains and 60,000 AI answers to see whether an exact-match domain still helps you rank or get cited by AI search.

Where In An Article Do AI Engines Pull Citations From?

Nov 2, 2026 · 13 min read

Where In An Article Do AI Engines Pull Citations From?

Where in an article AI engines pull citations from, from the CXL, Ahrefs, Previsible and Princeton studies: the top third, self-contained blocks, headings, length, and where the data disagrees.

What Is An AI Citation Engine?

Oct 30, 2026 · 9 min read

What Is An AI Citation Engine?

What an AI citation engine is, how ChatGPT, Perplexity and Google decide which pages to cite, why most pages they read never get cited, and how to measure it.

Reddit, Reviews, And Brand Mentions: What AI Search Engines Trust Now

Oct 30, 2026 · 10 min read

Reddit, Reviews, And Brand Mentions: What AI Search Engines Trust Now

What AI search engines trust in 2026: how ChatGPT, Gemini, Perplexity and Claude weigh Reddit threads, review sites, press mentions and your own website.

Why AI Visibility Isn't One Number: Per-Engine Citation Data

Oct 30, 2026 · 11 min read

Why AI Visibility Isn't One Number: Per-Engine Citation Data

AI visibility statistics by engine for 2026: the sites each AI assistant cites most, how little their sources overlap, and how fast their citations fade.

How Does ChatGPT Decide Which Websites To Recommend?

Sep 25, 2026 · 8 min read

How Does ChatGPT Decide Which Websites To Recommend?

The two-system answer: model memory vs live search, the retrieval path, the third-party pages that actually decide recommendations, and how to move each signal.

Does Blocking AI Crawlers Stop Perplexity From Citing Your Content?

Sep 19, 2026 · 9 min read

Does Blocking AI Crawlers Stop Perplexity From Citing Your Content?

Mostly no. Blocking PerplexityBot removes you from its index, but on-demand fetches and third-party paths remain. What blocking actually changes.

THE SIGNAL

Insights on AI and Google search strategy. No fluff.

Get the latest on AI citations, Google rankings, and content strategy.

No spam. Unsubscribe anytime.