Your robots.txt can be perfect in the repo and wrong on the live site. A dev team can burn a week polishing it and stay invisible in Claude, because the bots download a different version, and most "we optimized for AI search but nothing moved" stories end in that gap.
You can't treat this file as set-and-forget now. Every LLM vendor ships two or three crawlers, each with its own job, and unless you can read the syntax cold you can't debug what your site is saying to GPTBot, ClaudeBot, PerplexityBot and Google-Extended. This post covers the directives themselves. The AI crawler checklist has the 30-minute setup, and the complete AI crawler list goes through the bots vendor by vendor.
RFC 9309: what the spec actually says
The rule I'd want every developer to know by heart covers user-agent tokens. A product token can only contain letters, underscores and hyphens, which rules out wildcards completely: User-agent: GPT* has undefined behavior, and each vendor's bot has to be named on a User-agent line of its own. I list it first because a wildcard token looks sensible enough to pass review.
RFC 9309 only arrived in September 2022, when the IETF ratified the protocol twenty-eight years after it was first proposed, and any AI crawler that honors robots.txt honors this spec.
Four directives are formally defined. User-agent says which crawler or crawlers a group is for. Disallow shuts off a path or a pattern of paths, Allow opens a path back up inside a Disallowed tree, and # marks a comment through to the end of the line.
Past those four, you're outside the spec. Sitemap gets an informal mention, and most crawlers read it anyway. Crawl-delay isn't in the RFC at all: Google explicitly ignores it, though Bing, Yandex, Anthropic and most AI crawlers respect it as a de-facto extension. A leftover Host line can go.
Per-vendor tokens: the directive map
Copy from this table character for character. Each entry is a user-agent string in live use as of July 2026, and one misplaced hyphen means the crawler never matches your directive. Unless a row's notes say otherwise, assume the bot follows RFC 9309.
| Vendor | Token | Purpose |
|---|---|---|
| OpenAI | GPTBot | Training data crawl for GPT |
| OpenAI | OAI-SearchBot | ChatGPT Search indexing |
| OpenAI | ChatGPT-User | User-triggered browsing/Actions |
| OpenAI | OAI-AdsBot | ChatGPT ads landing-page validation |
| Anthropic | ClaudeBot | Training data crawl |
| Anthropic | Claude-User | Interactive agent fetches |
| Anthropic | Claude-SearchBot | Search indexing for Claude |
Googlebot | Standard Search crawl | |
Google-Extended | Gemini/Vertex AI training opt-out (control token) | |
GoogleOther / GoogleOther-Image / GoogleOther-Video | Experimental/AI use crawls | |
| Perplexity | PerplexityBot | Indexing and training (violations confirmed) |
| Perplexity | Perplexity-User | Real-time answer fetches (violations confirmed) |
| Microsoft | Bingbot | Bing Search index and Copilot |
| Apple | Applebot | Siri, Spotlight, Safari Suggestions |
| Apple | Applebot-Extended | Apple Intelligence training opt-out |
| Meta | meta-externalagent | Meta AI training and indexing |
| Meta | meta-externalfetcher | User-initiated Meta AI fetches |
| ByteDance | Bytespider | TikTok LLM training (documented non-compliance) |
| Amazon | Amazonbot | Alexa and Amazon AI training |
| Common Crawl | CCBot | Open dataset feeding GPT, Llama, Mistral, more |
| Cohere | cohere-ai | Cohere Command/Embed training |
| Mistral | MistralAI-User / MistralAI-Index | Vibe assistant fetches and indexing |
| DuckDuckGo | DuckAssistBot | DuckAssist AI answers |
| You.com | YouBot | You.com AI search |
| Exa | Exa-Bot | Exa AI search indexing |
| xAI / Grok | (none honored) | No token appears in HTTP requests |
Two rows are worth slowing down for. Google-Extended is a control token, and you'll never see it in an HTTP User-Agent header, because the requests come from Googlebot and GoogleOther. Blocking it asks Google to keep what those crawlers collect out of Gemini training, and your Search rankings don't move.
The other is the split by job. OpenAI runs GPTBot and OAI-SearchBot side by side: keep GPTBot out and your pages stay out of GPT training, keep OAI-SearchBot out and ChatGPT Search stops citing you. Anthropic's ClaudeBot, Claude-User and Claude-SearchBot split the same way, and so do Applebot and Applebot-Extended, which is why each one needs a separate entry.
Apple updates its Applebot documentation explaining what it means to block Applebot-Extended vs Applebot https://t.co/IUUUslM49j https://t.co/7tFeBLPzvf
Barry Schwartz@rustybrickApr 28, 2025
When Apple clarified its own split, Barry Schwartz at Search Engine Roundtable flagged it. Siri, Spotlight and Safari suggestions all run on Applebot, and Applebot-Extended only decides whether Apple Intelligence may train on your pages. So Disallow: / on Applebot removes you from Siri, while the identical line on Applebot-Extended leaves your Siri visibility where it was.
The first-match trap: how group precedence works
If you're short on review time, spend it here. AI crawlers follow these rules to the letter, and they seem obvious until a CDN edits your file.
Say the file contains both User-agent: * and User-agent: GPTBot. GPTBot follows only its named group, because a named group always beats *, and since groups never inherit from one another, each starts empty. To give several crawlers a single block, stack their User-agent lines before the first rule and they share the group.
Inside a group, the rule with the longest matching path applies. That's why Allow: /blog/press-kit/ outranks Disallow: /blog/ for a URL under press-kit, and when two paths tie on length, the Allow wins.
The Cloudflare problem hides in that first rule. Its Managed robots.txt put AI-blocking directives at the top of the file, ahead of the customer's own rules, and the first group to match is the one that counts:
Cloudflare's "Managed robots.txt" silently blocks GPTBot/ClaudeBot — anyone else seeing this?
Working in SEO/GEO for Turkish market 6+ years. Last year clients started reporting their sites stopped appearing in ChatGPT/Claude answers, even after doing all the obvious GEO work (llms.txt, schema markup, etc.). After investigating, I f...
A Turkish SEO practitioner surfaced the case while working out why client sites had dropped out of ChatGPT and Claude answers. Cloudflare's lines sat on top, so under first-match every Allow the owner had typed below them never applied. Your CMS shows the file you wrote, so curl the live URL and read what a bot gets.
We'll show you exactly where your brand stands in AI search.
No commitment. $0 due today, cancel anytime. See how ChatGPT, Perplexity, Claude, Gemini, Grok, and Google AI Mode talk about your brand today.
Wildcards, anchors and path patterns
Paths get a mini-syntax of their own, built on just two characters, * and $.
| Pattern | Meaning | Example |
|---|---|---|
/path/ | Prefix match | Disallow: /admin/ |
/path/*.ext$ | Files ending in .ext under path | Disallow: /uploads/*.pdf$ |
/*.php | Any .php URL anywhere | Disallow: /*.php |
/page$ | Only /page, not /page-2 | Disallow: /page$ |
Disallow: (empty) | Explicit all-allow for the group | Disallow: |
I'd get comfortable with $ first, since nothing else pins a rule to one exact URL. Without it, Disallow: /page catches /page-2, /page/subpage and /pages/ as well, plus any other URL that begins the same way. A trailing * is the opposite case: it adds nothing, and /api/* matches exactly what /api/ does.
Escape hatches: Allow, noai and X-Robots-Tag
To keep a directory closed except for one slice of it, Allow is my first tool. The longer path wins, so nesting a longer Allow inside a shorter Disallow gives you a legal exception:
User-agent: GPTBot
Disallow: /blog/
Allow: /blog/press-kit/
HTML pages get a second layer through the robots meta tag, where noai and noimageai ask that the page stay out of AI training and AI image generation. Nobody is obliged to listen. OpenAI, Anthropic and Adobe honor the tags for images, plenty of other crawlers don't, so use them alongside robots.txt and never in its place.
For files that aren't HTML, the same directives travel in the X-Robots-Tag response header. There's no way to put a <meta> tag inside a PDF, a feed or a binary file, which makes a header like X-Robots-Tag: noai, noimageai the only exit those files have.
The confusion I'd most like to spare you is Disallow versus noindex. A Disallow means the page is never requested. A noindex only works after the fetch, when the crawler reads the tag and leaves the URL out of the index. Disallow a URL and your noindex is never read, so the URL can still appear in search as a bare title with a link. Keep Disallow for content no crawler should touch, and noindex for pages that may be crawled but not indexed.
Where robots.txt alone falls short
A file that asks politely only stops the crawlers that listen, so I'd never make robots.txt the only barrier between your content and a scraper. Perplexity is why.
In August 2025 it was caught hitting sites that had blocked it in robots.txt and with WAF rules. Cloudflare followed the requests back to Perplexity IPs cycling through browser-like user-agent strings. Cloudflare took Perplexity off its Verified Bot list, and later that year Reddit and The New York Times sued in federal court, alleging DMCA anti-circumvention.
Cloudflare changed its own defaults on September 15, 2026. It now groups AI crawlers by what they do, into Search, Agent and Training, and on ad-monetized pages of new domains the Training and Agent crawlers are blocked out of the box, whatever the site's robots.txt allows.
Cloudflare just reshaped how AI crawlers get classified — and SEO teams need to pay attention before Sept 15. On its second "Content Independence Day," Cloudflare moved away from the blunt "block all AI bots" model toward a behaviour-based taxonomy. Instead of asking "is this
Pedro Dias@pedrodiasJul 2, 2026Pedro Dias, an independent analyst, summed up the change well: infrastructure is moving away from the question "is this an AI bot?" and toward judging bots by behavior. A User-agent line can only match the name a bot gives, which leaves robots.txt a step behind.
A few bots sit outside the file altogether. Grok, from xAI, sends no published bot token in its HTTP requests, so robots.txt has nothing to match. And ai.txt and llms.txt won't cover the gap, since they're adjacent formats and neither can block a crawler.
Know exactly what AI says about your competitors.
RankControl's Recon Agent monitors competitor citations across ChatGPT, Perplexity, Claude, Gemini, Grok, and Google AI Mode. See where they show up and you don't.
A sensible SaaS robots.txt for 2026
For a typical SaaS site, my starting file lets in the crawlers that lead to citations and turns away the training pipelines that return nothing. Admin paths stay closed to every bot. Read the comments before you paste.
# robots.txt — SaaS defaults, 2026
# Protected paths
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
# OpenAI: allow search/user, block training
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
# Anthropic: allow search/user, block training
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
# Google: allow Search, block Gemini training
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
# Perplexity: allow, pair with WAF
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Apple: allow Siri, block Apple Intelligence training
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /
# Bing/Copilot, Mistral, DuckAssist, You, Exa
User-agent: Bingbot
Allow: /
User-agent: MistralAI-User
Allow: /
User-agent: MistralAI-Index
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: YouBot
Allow: /
User-agent: Exa-Bot
Allow: /
# Meta: block training crawler, allow user fetches
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Allow: /
# Training-only bots: block
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Amazonbot
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xml
Any of the training-versus-search choices can be flipped. A subscription business that wants every citation it can get will find this shape easy to defend. If what you publish is proprietary research that should never be reused, I'd let only the Search bots in and add WAF rules at the CDN to catch spoofed bots.
Common syntax mistakes that break everything
When a robots.txt breaks, it's almost always one of a handful of errors, so check yours against this table first.
| What the file says | What goes wrong |
|---|---|
User-agent: Claude-Bot | Wrong token: it's ClaudeBot, one word |
User-agent: ClaudeUser | Wrong token: it's Claude-User, hyphenated |
User-agent: GPT* | Undefined: wildcards inside a User-agent token are invalid |
User-agent: Google-Extended with no Allow or Disallow after it | An empty group does nothing |
Google-Extended treated like a crawler UA in log analysis | It never appears in logs; Googlebot does the fetch |
| Reordering lines to change precedence | Order doesn't matter: the longest path wins wherever it sits |
The one I'd lose sleep over is quieter: a file that looks right in the repo and gets replaced before any crawler sees it. Your traffic won't show it, because citation crawlers send back so little. One GEO analyst went through Cloudflare's data and put Anthropic's crawl-to-referral ratio at roughly 100,000 to 1:
Your robots.txt is probably making you invisible to ChatGPT
Been digging into this for a week and the numbers are bad enough that I think most agencies haven't actually checked. 1. The crawl-to-refer ratio problem Cloudflare published data showing how often AI bots crawl a site vs. how often they se...
Numbers like that make every block look free, and for a crawler that only trains, it is. Turning away a citation crawler costs you nothing in referrals either, yet it decides whether Claude's answers can include your brand at all.
Keeping this right by hand means curling your live URL every quarter and diffing it against the file you think is deployed. Or let RankControl check every day whether AI search crawlers can reach your site; it alerts you when robots.txt or your CDN blocks them.
26 content formats. Published on your domain. Matched to your brand.
Guides, comparisons, listicles, case studies, and more. RankControl generates content that gets cited by ChatGPT, Perplexity, Claude, Gemini, Grok, and Google AI Mode.
Write the file carefully, then look at what the bots are served. Teams that skip that look keep wondering why Claude can't see their content, when the cause is five lines someone prepended above their hand-written rules.
