Robots.txt was invented in 1994 as a gentleman's agreement, and the developer community has spent 2026 discovering that AI crawlers do not do gentleman's agreements. Someone on r/neocities ran the cleanest test of the year:
so AI crawlers just ignore your robots.txt and apologize when confronted lol
my company gives me access to claude code for work and I tried to see if it will respect my robots.txt that explicitly blocks ai crawlers and what not and surprise surprise, they really don't care lol I'm thinking of starting to poison all...
Claude Code fetched a page that robots.txt explicitly forbade. The apology was polite. The fetch already happened. Three hundred and eighty-seven upvotes worth of developers already knew, and one of them named the whole discipline in one line: robots.txt is a good-will band-aid solution. Which is why knowing the actual list of AI crawlers active on the web in 2026, with the user-agent strings, IP ranges, robots.txt directives, and compliance patterns, is table stakes now. And it matters more this quarter because Cloudflare's default rules flip on September 15.
This guide is the reference. Bookmark it. There are 31 named AI crawlers from 16 vendors active as of July 2026 that this article documents by name, with links to the official docs for each. Section by section, here they are.
The Three-Category Framework
Cloudflare's classification is the one to internalize, because it's the vocabulary the rest of the ecosystem is settling on. Every AI crawler fits into one of three categories:
- Training: crawls the web to feed foundation-model pretraining or fine-tuning. Examples: GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot.
- Search: indexes content to answer user questions later, similar to how Bingbot works. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index.
- Agent: real-time fetches on behalf of a specific user who asked a specific question. Examples: ChatGPT-User, Claude-User, Perplexity-User, Google-Agent, Meta-ExternalFetcher.
The important part: many vendors run one bot per category, and a robots.txt directive that blocks Training will not affect the Search or Agent bots from the same company. Blocking ClaudeBot does not block Claude-User. Blocking GPTBot does not block ChatGPT-User. The three-bot pattern (Training, Search, Agent) is now the norm from the four largest AI vendors, so choose your policy per category, not per company.
The independent SEO analyst Pedro Dias wrote the cleanest explainer of Cloudflare's taxonomy last week, worth reading if you want the deep version. Now the bots themselves.
OpenAI (4 Bots)
Docs: platform.openai.com/docs/bots. All four respect robots.txt per OpenAI's own documentation. IP ranges published as JSON files updated regularly.
| Bot | Category | User-Agent Token | IP Ranges |
|---|---|---|---|
| GPTBot | Training | GPTBot/1.4 | openai.com/gptbot.json |
| ChatGPT-User | Agent (user-triggered) | ChatGPT-User/1.0 | openai.com/chatgpt-user.json |
| OAI-SearchBot | Search | OAI-SearchBot/1.4 | openai.com/searchbot.json |
| OAI-AdsBot | Ad landing page validator | OAI-AdsBot/1.0 | openai.com/adsbot.json |
OAI-AdsBot was added to the OpenAI docs in April 2026 and is a bot most coverage still misses. It checks landing pages for ChatGPT ad policy compliance and doesn't feed training data. Safe to allow unless you never plan to run ChatGPT ads.
Anthropic (3 Bots)
Docs: support.claude.com/en/articles/8896518. IP list at claude.com/crawling/bots.json (20 IPv4 prefixes as of May 2026). All three respect robots.txt per Anthropic's docs.
| Bot | Category | User-Agent Token | Notes |
|---|---|---|---|
| ClaudeBot | Training | ClaudeBot/1.0 | Content may train Claude foundation models |
| Claude-User | Agent | Claude-User | Real-time fetch when a Claude user asks a question |
| Claude-SearchBot | Search | Claude-SearchBot/1.0 | Indexes for Claude's retrieval layer, not training |
The three-bot framework was clarified in an Anthropic docs update in February 2026 that a lot of teams missed.
Your competitors are building backlinks while you read this.
Organic outreach, social mentions, exchanges, and done-for-you link building. Four ways to grow your domain authority on autopilot.
Google (8 Crawlers and Tokens)
Docs: developers.google.com/search/docs/crawling-indexing/overview-google-crawlers. IP list at developers.google.com/static/crawling/ipranges/common-crawlers.json. All respect robots.txt except Google-Agent (user-triggered fetcher).
| Bot | Category | Token |
|---|---|---|
| Googlebot | Search (index) | Googlebot |
| Googlebot-Image | Search (image index) | Googlebot-Image |
| Google-Extended | Training opt-out token | Google-Extended |
| GoogleOther | Research/product crawling | GoogleOther |
| GoogleOther-Image | Research/product image crawling | GoogleOther-Image |
| Google-CloudVertexBot | Owner-requested Vertex AI Agent build | Google-CloudVertexBot |
| Google-Agent | Agent (user-triggered) | Google-Agent |
Google-Extended is a token, not a separate crawler. It piggybacks on Googlebot requests and controls whether the crawled data trains Gemini or Vertex. Blocking Google-Extended opts you out of AI training without hurting your Google Search ranking. Google-Agent shipped in March 2026 and replaced the retired GoogleAgent-Mariner (Project Mariner shut down May 4, 2026).
Perplexity (2 Bots)
Docs: docs.perplexity.ai/docs/resources/perplexity-crawlers. IP lists at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
| Bot | Category | User-Agent Token |
|---|---|---|
| PerplexityBot | Search | PerplexityBot/1.0 |
| Perplexity-User | Agent | Perplexity-User/1.0 |
Cloudflare accused Perplexity of ignoring robots.txt in 2025 by rotating IPs and identities. Perplexity's official docs say PerplexityBot respects robots.txt; Perplexity-User is documented as user-triggered and does not.
Meta (3 Bots)
Docs: developers.facebook.com/docs/sharing/webmasters/crawler.
| Bot | Category | User-Agent Token |
|---|---|---|
| Meta-ExternalAgent | Training | meta-externalagent/1.1 |
| Meta-ExternalFetcher | Agent | meta-externalfetcher/1.1 |
| FacebookBot (legacy) | Training (speech) | facebookexternalhit/1.1 |
Cloudflare Radar ranks Meta-ExternalAgent as the second highest volume bot on the web after Googlebot, ahead of GPTBot. If your bandwidth is under stress, this is the one to look at first.
Microsoft (2 Bots)
Microsoft Copilot has no dedicated user-agent as of July 2026. It uses Bingbot infrastructure. Blocking Bingbot will affect both Bing Search and Copilot answers.
| Bot | Category | Token |
|---|---|---|
| Bingbot | Search (also feeds Copilot) | bingbot/2.0 |
| MSNBot | Legacy | msnbot |
ByteDance and Amazon and Apple
ByteDance / Bytespider. Training crawler for Doubao and TikTok. Cloudflare and multiple security researchers document Bytespider ignoring robots.txt and using disguised user-agents. Block at the CDN layer; robots.txt on its own won't hold.
Amazon publishes three bots at developer.amazon.com/amazonbot:
| Bot | Category | Token |
|---|---|---|
| Amazonbot | Training (Alexa) | Amazonbot/0.1 |
| Amzn-SearchBot | Search | Amzn-SearchBot/0.1 |
| Amzn-User | Agent | Amzn-User/0.1 |
Reverse DNS verification is Amazon's recommended check. Amazon bots do not support the Crawl-delay directive.
Apple. Applebot is the primary crawler (search indexing for Siri, Spotlight, Apple Intelligence). Applebot-Extended is a token that controls whether Applebot-crawled data trains Apple's generative AI, same pattern as Google-Extended. Docs at support.apple.com/en-us/119829. IP list at apple.com/go/applebot (12 IPv4 CIDRs as of July 2026).
Everyone Else, in Brief
The long tail matters more than most guides admit. Every one of these bots was verified against official docs where available:
- Common Crawl / CCBot: respects robots.txt; fake CCBot impersonators are common. Verify via reverse DNS against
*.commoncrawl.org. - Mistral: MistralAI-User (Agent) and MistralAI-Index (Search). Docs at docs.mistral.ai/robots. IP JSON published.
- Cohere: cohere-ai and cohere-training-data-crawler (both Training). Documented compliance.
- DuckDuckGo / DuckAssistBot: retrieval agent for DuckDuckGo AI answers. Respects robots.txt.
- You.com / YouBot: search index and retrieval. Publishes HTTP Message Signatures directory.
- Alibaba / Qwen: QwenBot (Training), TongyiBot (indexing), Qwen-User (Agent). English docs are sparse.
- Huawei / PetalBot: search index for Petal Search. Respects robots.txt.
- Brave / Bravebot: search index. No official verification method published.
- Exa / ExaBot: semantic search indexing for LLM apps.
- ImagesiftBot: image discovery for ImageSift (Hive).
- Diffbot: commercial structured-data extraction. Respects robots.txt by default; customer-configurable.
The xAI / Grok Problem
xAI publishes no official crawler documentation page as of July 2026. Behavioral analysis by multiple independent researchers documents Grok crawlers using residential IP rotation, spoofed Chrome and Safari user-agents, and Go-http-client strings. One query has been documented triggering sixteen requests from twelve IPs impersonating human browsers. The nominal GrokBot/1.0 and xAI-Grok/1.0 tokens exist but are largely window dressing. Robots.txt directives are not a reliable control against Grok traffic; blocking has to happen at the CDN based on behavioral signatures.
Cloudflare's September 15 Rule Change
Starting September 15, 2026, Cloudflare's default bot policies change. The new defaults will block Training and Agent bots on ad-displaying pages for new customers, all free-tier existing customers, and any new domains added by paid customers after that date. Search bots remain allowed by default. Mixed-purpose crawlers that combine Search and Training are treated as Training if you block Training, which is the source of the biggest self-inflicted-injury risk this month:
Is your site sitting behind Cloudflare? If yes, Go to Security -> Settings -> Configure AI bot policies Under "Training" section make sure "Allow (do not block)" is selected. Do this before 15th of September. Why? From this date onwards, If you have selected the block https://t.co/bBH2zwLHjw
Suganthan Mohanadasan@suganthanJul 3, 2026
Read that one twice if you're on Cloudflare. Blocking training without checking the mixed-purpose flag will accidentally block Googlebot, Applebot, and Bingbot, which is a fast way to wipe out your regular search traffic. Sept 15 is a settings-audit deadline, not a policy declaration.
The community reaction thread lays out the debate honestly:
Cloudflare is about to block AI agents by default on a fifth of the web. Nobody I talk to outside of tech knows this is coming. Why is no one talking about it?
Starting September 15, Cloudflare’s new AI traffic controls will split bots into Search, Agent, and Training, blocking agents and training bots by default on any page that shows an ad, while letting search bots through. That machine web is...
Signed Agents (Web Bot Auth) is the underlying identity infrastructure Cloudflare wants everyone on. It uses HTTP Message Signatures (RFC 9421), Ed25519 keys, and a JWKS directory at a well-known URL. Bots that sign requests correctly get through automatic bot policies without needing verified-bot allowlists. First adopters as of March 2026: ChatGPT Agent, Goose (Block), Browserbase, Anchor Browser, and Cloudflare's own Browser Rendering.
You're getting AI traffic. But are you capturing the leads?
RankControl tracks every visitor from ChatGPT, Perplexity, and Claude. Full source attribution, intent scoring, and A/B tested lead capture.
What Publishers Are Actually Doing
Not a mystery, and worth calibrating your own policy against the data:
79% of world’s biggest news publishers now block AI training bots "79% of 100 top news websites in the UK and US are blocking at least one crawler used for AI training out of OpenAI’s GPTBot, ClaudeBot, Anthropic-ai, CCBot, Applebot-Extended and Google-Extended." "Meanwhile https://t.co/mIAlm5hPqR
Glenn Gabe@glenngabeJan 22, 2026
Seventy-nine percent of top news publishers block at least one AI training bot; seventy-one percent block retrieval bots too. Which is the choice you can make honestly: opt out of training and lose the pretraining share, opt in and let the models eat your archive. Both are legitimate calls. The one that isn't legitimate anymore is not knowing which bots are hitting your site.
Recommended Defaults for a SaaS Site
A concise policy that survives September 15 without over-blocking:
- Allow all Search bots, because they can send referral traffic. That includes OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index, YouBot, DuckAssistBot, Amzn-SearchBot, Bingbot, Googlebot, and Applebot.
- Allow all Agent bots, because they're user-triggered and the traffic tends to convert. That includes ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Google-Agent, Amzn-User, MistralAI-User.
- Decide on Training bots case by case. Block GPTBot and ClaudeBot if your legal team says so; allow if your content strategy wants to be sampled by foundation models. Google-Extended and Applebot-Extended are token-level controls that let you opt out of training without hurting search.
- Always block Bytespider and any Grok-style scraper at the CDN. Robots.txt won't stop them.
- Never rely on user-agent alone. Verify against published IP ranges or reverse DNS. Impersonators are common and cheap.
Then run the audit against your Cloudflare or CDN dashboard once a month. The Ahrefs Bot Analytics tool surfaces the same data at the request level if you want to see it inside a familiar SEO interface. Our own AI visibility tracking closes the loop by measuring whether crawler traffic translates into citations across engines, and the llms.txt and robots.txt setup checklist covers the specific config snippets for the biggest four bots in the 30 minutes it takes to ship them.
We'll show you exactly where your brand stands in AI search.
No commitment. No credit card. See how ChatGPT, Perplexity, Claude, and Gemini talk about your brand today.
The list will grow again before the next Google I/O. Bookmark this page or the equivalent one from Cloudflare's Radar bot directory, because the six months from launch to obscurity for a new AI crawler is the shape of 2026, and settling on a per-category policy now beats chasing the next Qwen-DeepResearch-User-Agent-v2 in the middle of a Friday afternoon incident.
