If someone told you robots.txt keeps AI crawlers out, you were sold a 1994 gentleman's agreement, and this year developers keep learning that AI crawlers don't do gentleman's agreements. One r/neocities user ran the most convincing test you'll find:
so AI crawlers just ignore your robots.txt and apologize when confronted lol
my company gives me access to claude code for work and I tried to see if it will respect my robots.txt that explicitly blocks ai crawlers and what not and surprise surprise, they really don't care lol I'm thinking of starting to poison all...
Claude Code fetched a page the site's robots.txt said it couldn't have. The apology that followed was polite, and it changed nothing, because the page was already fetched. Developers had seen this coming, judging by the 387 upvotes, and one of them put the whole practice in a phrase: robots.txt is a good-will band-aid solution.
So if you manage a site, you need the real list: each AI crawler's user-agent string and IP ranges, the robots.txt directives it responds to, and whether it complies. You need it sooner this quarter, because Cloudflare flips its default rules on September 15.
Keep this page bookmarked. It names 31 AI crawlers from 16 vendors that were active as of July 2026 and links each one to its official docs, vendor by vendor.
The three-category framework
Before you write a single rule, sort the bots the way Cloudflare does, since the rest of the industry is adopting its vocabulary:
- Training bots crawl the web to feed pretraining or fine-tuning of foundation models. GPTBot, ClaudeBot, Meta-ExternalAgent and CCBot are training bots.
- Search bots index content now so they can answer user questions later, much as Bingbot does. OAI-SearchBot, Claude-SearchBot, PerplexityBot and MistralAI-Index belong here.
- Agent bots fetch pages in real time for one user who asked one question. ChatGPT-User, Claude-User and Perplexity-User work this way, and so do Google-Agent and Meta-ExternalFetcher.
Why does the split matter to you? Most vendors now run a separate bot for each category, and your robots.txt rules apply per bot. Block ClaudeBot and you've done nothing about Claude-User; block GPTBot and ChatGPT-User still gets through. The four largest AI vendors all run this three-bot setup, so write your policy one category at a time rather than one company at a time.
If you want to go deeper, Pedro Dias, the independent SEO analyst, wrote the clearest explanation of Cloudflare's taxonomy last week. Next come the bots, vendor by vendor.
OpenAI (four bots)
Which OpenAI bot are you actually blocking? There are four, all listed at platform.openai.com/docs/bots, and OpenAI's documentation says every one of them respects robots.txt. Their IP ranges come as JSON files that OpenAI updates regularly.
| Bot | Category | User-Agent Token | IP Ranges |
|---|---|---|---|
| GPTBot | Training | GPTBot/1.4 | openai.com/gptbot.json |
| ChatGPT-User | Agent (user-triggered) | ChatGPT-User/1.0 | openai.com/chatgpt-user.json |
| OAI-SearchBot | Search | OAI-SearchBot/1.4 | openai.com/searchbot.json |
| OAI-AdsBot | Ad landing page validator | OAI-AdsBot/1.0 | openai.com/adsbot.json |
If you run ChatGPT ads, or might one day, leave OAI-AdsBot alone. It showed up in OpenAI's docs in April 2026, most coverage still skips it, and all it does is check that your landing pages follow ChatGPT's ad policies. None of what it reads feeds training data.
Anthropic (three bots)
If you set up your Anthropic rules before February 2026, look at them again. A docs update that month clarified the three-bot split, and a lot of teams missed it. The docs are at support.claude.com/en/articles/8896518, and the IP list at claude.com/crawling/bots.json held 20 IPv4 prefixes in May 2026. Anthropic says all three bots respect robots.txt.
| Bot | Category | User-Agent Token | Notes |
|---|---|---|---|
| ClaudeBot | Training | ClaudeBot/1.0 | Content may train Claude foundation models |
| Claude-User | Agent | Claude-User | Real-time fetch when a Claude user asks a question |
| Claude-SearchBot | Search | Claude-SearchBot/1.0 | Indexes for Claude's retrieval layer, not training |
Your competitors are building backlinks while you read this.
Organic outreach, social mentions, and link exchanges, with managed backlinks available as an add-on. Grow your domain authority without running the campaign yourself.
Google (eight crawlers and tokens)
Worried that blocking AI training will cost you Google rankings? It won't, as long as you use Google-Extended. That's a token, and no separate crawler sits behind it: it rides on Googlebot's requests and tells Google whether the crawled data may train Gemini or Vertex. Block it and you're out of AI training while your Search ranking stays where it is.
Google's crawler documentation is at developers.google.com/search/docs/crawling-indexing/overview-google-crawlers, with the IP list at developers.google.com/static/crawling/ipranges/common-crawlers.json. Every crawler here respects robots.txt apart from Google-Agent, the one users trigger.
| Bot | Category | Token |
|---|---|---|
| Googlebot | Search (index) | Googlebot |
| Googlebot-Image | Search (image index) | Googlebot-Image |
| Google-Extended | Training opt-out token | Google-Extended |
| GoogleOther | Research/product crawling | GoogleOther |
| GoogleOther-Image | Research/product image crawling | GoogleOther-Image |
| Google-CloudVertexBot | Owner-requested Vertex AI Agent build | Google-CloudVertexBot |
| Google-Agent | Agent (user-triggered) | Google-Agent |
Google-Agent is also the newest name on that list. It shipped in March 2026 as the replacement for GoogleAgent-Mariner, which was retired when Project Mariner shut down on May 4, 2026.
Perplexity (two bots)
If Perplexity's reputation makes you nervous, the record is mixed. Cloudflare accused it in 2025 of dodging robots.txt by rotating IPs and identities. By Perplexity's own docs, PerplexityBot respects robots.txt, while Perplexity-User is user-triggered and doesn't. You'll find the docs at docs.perplexity.ai/docs/resources/perplexity-crawlers and the IP lists at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
| Bot | Category | User-Agent Token |
|---|---|---|
| PerplexityBot | Search | PerplexityBot/1.0 |
| Perplexity-User | Agent | Perplexity-User/1.0 |
Meta (three bots)
Is your bandwidth bill climbing? Check Meta-ExternalAgent first. Cloudflare Radar ranks it the second-busiest bot on the web, behind Googlebot and ahead of GPTBot. Meta's crawler documentation is at developers.facebook.com/docs/sharing/webmasters/crawler.
| Bot | Category | User-Agent Token |
|---|---|---|
| Meta-ExternalAgent | Training | meta-externalagent/1.1 |
| Meta-ExternalFetcher | Agent | meta-externalfetcher/1.1 |
| FacebookBot (legacy) | Training (speech) | facebookexternalhit/1.1 |
Microsoft (two bots)
Thinking of blocking Bingbot to keep Copilot out? You'd lose Bing Search along with it. As of July 2026, Copilot has no user-agent of its own and runs on Bingbot's infrastructure, so the two go together.
| Bot | Category | Token |
|---|---|---|
| Bingbot | Search (also feeds Copilot) | bingbot/2.0 |
| MSNBot | Legacy | msnbot |
ByteDance and Amazon and Apple
You don't need to deliberate over Bytespider, ByteDance's crawler that gathers training data for Doubao and TikTok. Security researchers, and Cloudflare too, have caught it ignoring robots.txt and hiding behind disguised user-agents, so block it at your CDN, where you don't depend on it obeying anything.
Amazon documents three bots at developer.amazon.com/amazonbot:
| Bot | Category | Token |
|---|---|---|
| Amazonbot | Training (Alexa) | Amazonbot/0.1 |
| Amzn-SearchBot | Search | Amzn-SearchBot/0.1 |
| Amzn-User | Agent | Amzn-User/0.1 |
If you rely on Crawl-delay to slow crawlers down, it won't work here, since Amazon's bots don't support it. Amazon's recommended way to verify them is reverse DNS.
Apple follows the same pattern as Google, so if you've set up Google-Extended, you already know how this works. Applebot is the main crawler, indexing for Siri, Spotlight and Apple Intelligence, and Applebot-Extended is the token that decides whether Applebot's crawl can train Apple's generative AI. You'll find the docs at support.apple.com/en-us/119829 and the IP list at apple.com/go/applebot, which held 12 IPv4 CIDRs as of July 2026.
Everyone else, in brief
You'll run into these smaller bots more often than most guides suggest, so don't skip them. Each was verified against official documentation where it exists:
| Vendor | Bots | Details |
|---|---|---|
| Common Crawl | CCBot | Respects robots.txt; fake CCBot impersonators are common, so verify via reverse DNS against *.commoncrawl.org |
| Mistral | MistralAI-User (Agent), MistralAI-Index (Search) | Docs at docs.mistral.ai/robots; IP JSON published |
| Cohere | cohere-ai, cohere-training-data-crawler (both Training) | Documented compliance |
| DuckDuckGo | DuckAssistBot | Retrieval agent for DuckDuckGo AI answers; respects robots.txt |
| You.com | YouBot | Search index and retrieval; publishes an HTTP Message Signatures directory |
| Alibaba / Qwen | QwenBot (Training), TongyiBot (indexing), Qwen-User (Agent) | English docs are sparse |
| Huawei | PetalBot | Search index for Petal Search; respects robots.txt |
| Brave | Bravebot | Search index; no official verification method published |
| Exa | ExaBot | Semantic search indexing for LLM apps |
| ImageSift (Hive) | ImagesiftBot | Image discovery for ImageSift |
| Diffbot | Diffbot | Commercial structured-data extraction; respects robots.txt by default, customer-configurable |
When you're reading logs, two rows need extra care: CCBot, because impersonators borrow its name, and Bravebot, because there's no official way to verify it.
The xAI / Grok problem
If you're hoping robots.txt will keep Grok out, it won't. As of July 2026, xAI hasn't published an official page documenting its crawlers at all. Several independent researchers who studied its behavior report Grok crawlers rotating residential IPs and faking user-agents from Chrome and Safari. Some of the requests show up as Go-http-client strings.
In one documented case, a single query set off sixteen requests from twelve IPs, every one of them posing as a human browser. Tokens called GrokBot/1.0 and xAI-Grok/1.0 exist in name only, as window dressing, so your CDN has to block Grok based on how its traffic behaves.
Cloudflare's September 15 rule change
Is your site behind Cloudflare? Then September 15, 2026 changes your defaults. On that date Cloudflare's default bot policies change for new customers, all existing free-tier customers and any new domain a paid customer adds afterward. Training and Agent bots get blocked by default on any page that carries ads, and Search bots stay allowed.
The part that can hurt you is the handling of mixed-purpose crawlers, the ones doing both Search and Training. Block Training and they're treated as Training too, which makes this month's biggest self-inflicted injury easy to cause:
Is your site sitting behind Cloudflare? If yes, Go to Security -> Settings -> Configure AI bot policies Under "Training" section make sure "Allow (do not block)" is selected. Do this before 15th of September. Why? From this date onwards, If you have selected the block https://t.co/bBH2zwLHjw
Suganthan Mohanadasan@suganthanJul 3, 2026
Read that twice. If you block training and skip the mixed-purpose flag, you can knock out Googlebot, Applebot and Bingbot by accident and lose your regular search traffic quickly. Use September 15 as your deadline to audit settings rather than as a reason to announce a policy.
Cloudflare's community thread has both sides of the argument:
Cloudflare is about to block AI agents by default on a fifth of the web. Nobody I talk to outside of tech knows this is coming. Why is no one talking about it?
Starting September 15, Cloudflare’s new AI traffic controls will split bots into Search, Agent, and Training, blocking agents and training bots by default on any page that shows an ad, while letting search bots through. That machine web is...
Behind all this is Signed Agents, or Web Bot Auth, the identity system Cloudflare wants every bot to adopt. A signed bot relies on HTTP Message Signatures (RFC 9421) with Ed25519 keys and puts a JWKS directory at a well-known URL. If its signatures check out, it passes automatic bot policies with no verified-bot allowlist required. As of March 2026, the first adopters were ChatGPT Agent, Goose (Block) and Browserbase, plus Cloudflare's own Browser Rendering and Anchor Browser.
You're getting AI traffic. But do you know where it comes from?
RankControl credits every visit to the assistant that sent it: ChatGPT, Perplexity, Claude, Gemini, Copilot, or Grok. Full source attribution, next to your Google traffic.
What publishers are actually doing
Wondering what's normal? Publishers give you a benchmark:
79% of world’s biggest news publishers now block AI training bots "79% of 100 top news websites in the UK and US are blocking at least one crawler used for AI training out of OpenAI’s GPTBot, ClaudeBot, Anthropic-ai, CCBot, Applebot-Extended and Google-Extended." "Meanwhile https://t.co/mIAlm5hPqR
Glenn Gabe@glenngabeJan 22, 2026
Of the top news publishers, seventy-nine percent block one or more AI training bots, and seventy-one percent also shut out retrieval bots. You face the same honest trade-off they did. Opt out of training and you lose your share of pretraining. Opt in and the models get your archive. Either is a legitimate call, but running a site without knowing which bots hit it no longer is.
Recommended defaults for a SaaS site
So what should your own policy look like? You want one that survives September 15 without blocking too much. Start by allowing every Search bot, because those can send you referral traffic, and every Agent bot, because they're user-triggered and that traffic tends to convert:
| Search bots to allow | Agent bots to allow |
|---|---|
| OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index, YouBot, DuckAssistBot, Amzn-SearchBot, Bingbot, Googlebot, Applebot | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Google-Agent, Amzn-User, MistralAI-User |
Training bots are your judgment call. If your legal team says to block GPTBot and ClaudeBot, block them. If your content strategy wants foundation models to sample your pages, allow them. Either way, you can use the Google-Extended and Applebot-Extended tokens to leave training while your search visibility stays intact.
Two rules don't bend. Bytespider and any Grok-style scraper get blocked at the CDN, since robots.txt won't stop them. And never trust a user-agent on its own: verify it with the vendor's published IP ranges or a reverse DNS lookup, because impersonators are common and cheap.
After that, compare your Cloudflare or CDN dashboard with your policy once a month. If you'd like the same data at request level in an SEO tool you already know, the Ahrefs Bot Analytics tool shows it. The AI visibility tracking we run closes the loop, because it measures whether that crawler traffic turns into citations across engines. For the config snippets covering the four biggest bots, about 30 minutes of work, use the llms.txt and robots.txt setup checklist.
We'll show you exactly where your brand stands in AI search.
No commitment. $0 due today, cancel anytime. See how ChatGPT, Perplexity, Claude, Gemini, Grok, and Google AI Mode talk about your brand today.
Expect more names on this list before Google I/O comes around again. Keep this page bookmarked, or Cloudflare's Radar bot directory, because a new AI crawler can launch and fade into obscurity within six months these days. You're better off settling a per-category policy now than hunting down the next Qwen-DeepResearch-User-Agent-v2 during some Friday-afternoon incident.
