On September 26 we pulled the robots.txt files from 20 well-known websites. Every one of the five news sites had rules that named AI crawlers. Of the other 15, only three mentioned an AI crawler by name at all. Two more sites, Vercel and Twilio, carried a line you've probably never seen: Content-Signal: search=yes, ai-input=yes, ai-train=no.
The file that stuck with me was HubSpot's. It's 12.6 KB long, with over 350 Disallow lines, and there's no AI bot anywhere in it.
So, do you block AI crawlers? For a business that sells something and wants to be found, mostly no. Leave AI search bots and user-triggered fetchers free to visit, and decide about training bots as a separate matter. Below I cover why, what the 20 sites actually do, and why a block often leaks even when you mean it.
AI crawlers come in three kinds
People say "AI crawler" as if it were one thing, but it covers three jobs, and each big vendor now runs a separate user agent per job:
| Kind | What it does | Examples | Block it and you lose |
|---|---|---|---|
| Training | Collects pages that may train future models | GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot | Nothing visible today |
| Search | Builds the index AI answers draw from | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your place in that engine's answers |
| User fetch | Opens a page because someone asked about it | ChatGPT-User, Claude-User, Perplexity-User | Little, since most of these ignore the block |

OpenAI lays the split out on its crawler page. You can let OAI-SearchBot in for search results while GPTBot stays disallowed. The page is blunt about the reverse case: "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Anthropic's three bots are split the same way, and Anthropic cautions that shutting out Claude-User could hurt how visible you are in user-directed search.
Two entries you'll find on most crawler lists don't crawl anything. Google-Extended is really a control token for Gemini training and grounding, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Applebot-Extended, Apple's version, "does not crawl webpages," so a block only keeps Apple from training on your pages.
For the full set of user agent strings, see our complete AI crawler list.
What blocking each kind costs you
Start with training bots, because a block there costs you very little that you'd notice. Your content just stops going into that vendor's future models, and your analytics look the same tomorrow as today. Over the long run there may be a cost you can't measure well: a model trained without your pages might have a thinner picture of your product when it answers from memory and not from a live search.
Search bots are the expensive ones to block. OpenAI's docs are unambiguous: block OAI-SearchBot and ChatGPT search answers won't include you. If you run a SaaS company, picture a buyer asking for the "best help desk for a 20-person team" and your product missing from the answer.
Blocking the user fetchers is mostly futile, and the vendors say as much. Per OpenAI, robots.txt rules "may not apply" to ChatGPT-User, because a person started the visit. Perplexity's docs go further and say Perplexity-User "generally ignores robots.txt rules." And Meta-ExternalFetcher, by Meta's own account, "may bypass robots.txt." With three vendors putting the same caveat in writing, I'd read robots.txt as a polite request that these fetchers can refuse.
What 20 big sites actually block
Our September 26 census, sorted by kind of site, looks like this. A site gets a yes under "Name an AI bot" when its robots.txt names at least one AI crawler in a rule, beyond the generic User-agent: * rules everyone has. Content-Signal is the newer preference line I'll get to further down.
| Group | Sites checked | Name an AI bot | Use Content-Signal | What stood out |
|---|---|---|---|---|
| News | NYT, WSJ, Guardian, Reuters, Washington Post | 5 of 5 | 0 | NYT blocks every AI bot we checked |
| SaaS | HubSpot, Salesforce, Zendesk, Atlassian, Intercom | 0 of 5 | 1 | Intercom's signal says yes to all three uses |
| Ecommerce | Nike, Target, Best Buy, Wayfair, Chewy | 1 of 5 | 0 | Best Buy files AI bots under "AI search and discovery bot access" |
| Developer tools | Stripe, GitHub, Vercel, Cloudflare, Twilio | 2 of 5 | 3 | Vercel and Twilio say no to training, yes to search |
GitHub has the most careful setup of the 20. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot each get a one-second crawl delay there. They can reach marketing pages such as /pricing and /features, but repository pages like commit histories and forks are off limits.
The news sites line up with a bigger study. In July, FT Strategies reviewed 70 large publishers. It found 39 of them fully blocking GPTBot, while just 16 blocked OAI-SearchBot. The authors see publishers shutting out training crawlers while taking "a more selective approach to those that feed AI-generated answers." They also admit the sample isn't representative, since the publishers were picked for being active in AI licensing debates.
One pattern in our census deserves a warning. At three of the five news sites, OpenAI's search and user bots got more access than their Anthropic or Perplexity counterparts. That probably reflects each publisher's business relationships more than any view on which bots behave, so don't copy a news site's robots.txt for your software company.

Your competitors are getting cited by AI. You're not.
Every day without citation tracking is a day your competitors pull ahead in ChatGPT, Perplexity, and Claude.
The leaky block
We also tested whether blocks hold up. Six of the 20 sites each got one question, asked both of GPT-5.5 with web search and of Perplexity's API. NYT, the Washington Post and Reuters block Perplexity's crawler, while Stripe, Zendesk and Nike block nothing.
The open sites were cited from their own pages, as you'd expect. Stripe's pricing page turned up, and so did Nike's returns policy and Zendesk's pricing page, although ChatGPT pointed to a Zendesk comparison page and not the pricing one.
Most of the blocked sites were cited too. PerplexityBot is blocked on the Washington Post's main domain, yet both engines cited pages from its help center and subscription subdomains, and those run their own robots.txt files. Reuters answers came from reutersagency.com, a different domain, plus one reuters.com article Perplexity cited. Perplexity went straight to nytimes.com subscription and help pages, despite the Times blocking every AI bot we looked at.
The name I've given this is the leaky block. The rule has a gap, whether that's a subdomain, a sibling domain or a user-triggered fetcher, and the content finds another way into AI answers through it. ChatGPT on the Times was the one block that held, and it cited a help portal run by a third party instead of nytimes.com.
Two caveats before you lean on any of this. We asked one question per site, so the check proves leaks happen and says nothing about how often. We also can't see how Perplexity reached the Times pages. It could have been its own crawler, a user fetch or an older copy. Cloudflare accused Perplexity in August 2025 of using a generic Chrome user agent to crawl sites that had blocked it, at three to six million requests a day, which Perplexity denied.
If blocking is your call, whether blocking stops Perplexity from citing you covers the enforcement layers.
Robots.txt is a request, and so is what replaces it
Cloudflare's Content Signals Policy, from September 2025, gives you a middle option. You add a Content-Signal line to robots.txt and answer three questions on their own: search indexing, live input for AI answers, and training. Both Vercel and Twilio set ai-train=no and leave the other two open. I suspect that's what most software companies want, and Disallow rules alone can't say it.

It's still only a preference. No crawler is obliged to honor the line, and the IETF group working on a standard for AI preferences puts technical enforcement outside its scope. In September, Search Engine Journal boiled down Helen Pollitt's advice: block at the highest layer you can reach, meaning a firewall or CDN, since a robots.txt rule only stops bots that decide to obey it.
Block AI crawlers in robots.txt or at the server level? Helen Pollitt's answer: Go as high up the stack as you can. WAF if you have one, CDN if you don't, server as a last resort. Robots.txt only works on bots that choose to respect it. More in this week's Ask An SEO 👇 https://t.co/XvVUAX7W13
SearchEngineJournal®@sejournalSep 3, 2026Blocking at the firewall comes with its own trap. One r/TechSEO poster turned on Cloudflare's AI training block in August and then saw Googlebot and Bingbot run into 403 errors on the sitemap. A commenter replied that Cloudflare documents this, since those bots count as multi-use crawlers there, and Google's John Mueller chimed in with an offer to look into it.
Has anyone else run into Cloudflare AI Crawlers blocking Googlebot/Bingbot?
I'm testing Cloudflare's new AI Crawlers & Scrapers feature and noticed something unexpected. When I set AI Training = Block, both Googlebot and Bingbot start receiving HTTP 403 responses when trying to fetch my sitemap. As soon as I disabl...
The default we'd pick for a business
If your company wants to be found, this is the robots.txt I'd start from:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Four training crawlers are blocked, and all the search bots and user fetchers stay open. Maybe you'd rather models learned from your pages, and plenty of software companies would. In that case, remove the four training blocks and flip ai-train to yes. Legal and marketing should decide that together, and either answer is fine provided the search bots remain open.
Once it's live, make sure it held. Go through the firewall or CDN settings for any AI switch that overrides robots.txt, and look at each subdomain that has its own file. The setup is in our robots.txt and llms.txt checklist, and a server log audit tells you which bots showed up after the change.
Plan on an hour for writing the file and going over the firewall settings. After that, about 30 minutes a month is enough to check that the bots you allowed keep visiting.
200+ SaaS teams already track their AI citations.
They know exactly when ChatGPT mentions their brand, and when it stops. Do you?

When blocking makes sense
The clearest case is a publisher whose business is the article itself. When an AI answer assembled from your reporting takes the place of a visit, blocking is a reasonable stance, and it strengthens your hand in licensing talks.
Over on r/webdev the complaint is different. In July a developer posted that a page-speed article of theirs had been scraped and put to use elsewhere within ten days, and the thread passed 200 upvotes. The useful advice in the replies was filing a DMCA request with Google over the copied URL, which, according to one commenter, usually gets scraped pages taken down in a couple of weeks. No robots.txt rule would have stopped that scraper, which never asked for permission.
Web scraping and content stealing is out of control
10 days. That's all it took for one of my articles on page speed to get scraped and reused in the marketing SEO slop of some online company, and then get found by me (purely by accident). I know it's my content because I think and write eve...
For everyone else, the bigger danger is a block you never meant to set, such as a CDN switch, a template you inherited or a newspaper's file you copied, that ends up locking out the search bots you need.
Watching what the bots do
You can do the upkeep by hand. Go back over robots.txt and the firewall whenever your CDN changes, read through your logs each month, and put your buyer questions to ChatGPT and Perplexity to make sure you're still being cited.
Or you can let RankControl do the watching. Our analytics track both AI crawler visits and AI-referred visits, which shows you the bots that still arrive after a change. The AI visibility tracking runs up to 50 buyer questions each week on seven AI engines. Start with the free AI visibility check to see where you stand today.
How often does ChatGPT mention your brand?
Most founders have no idea. The answer might surprise you.











