Robots.txt for AI: The Complete Directive Reference

A dev team can spend a week writing a beautiful robots.txt file and still be invisible in Claude, because the file they wrote is not the file that bots receive. That gap is where most "we optimized for AI search but nothing moved" stories end.

Robots.txt is no longer a file you set and forget. Every LLM vendor now ships two or three separate crawlers with separate purposes, and reading the syntax cold is the only way to debug what your site is actually telling GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. This piece is the directive reference. For the 30-minute setup workflow, see the AI crawler checklist. For the vendor-by-vendor bot inventory, see the complete AI crawler list.

RFC 9309: What the Spec Actually Says

The robots exclusion protocol was ratified by the IETF as RFC 9309 in September 2022. Twenty-eight years after the original proposal, robots.txt has a formal standard, and every AI crawler that respects the file respects this spec.

The RFC formally defines four directives:

• User-agent identifies the crawler(s) a group of rules applies to.
• Disallow blocks a path or path pattern.
• Allow permits a path within a Disallowed tree.
• # starts a comment that runs to end of line.

Sitemap is noted informally; most crawlers interpret it. Crawl-delay is not in the spec at all: Bing, Yandex, Anthropic, and most AI crawlers honor it as a de-facto extension, while Google explicitly ignores it. Host is safe to remove.

One rule catches people out. User-agent product tokens must contain only letters, underscores, and hyphens. Wildcards inside the token are forbidden. User-agent: GPT* is undefined behavior, and every vendor bot needs its own explicit User-agent line.

Per-Vendor Tokens: The Directive Map

Here is every token that matters as of July 2026. Each row is a real, current user-agent string. Copy them verbatim; a hyphen out of place drops the directive on the floor. Assume RFC 9309 compliance unless the notes say otherwise.

Two things jump out. First, Google-Extended is a control token, not a distinct crawler. It never appears in an HTTP User-Agent header; Googlebot and GoogleOther do the fetching, and the block signals "do not use crawled content to train Gemini." No Search ranking effect.

Second, GPTBot and OAI-SearchBot sit in the same account but do different jobs. Block GPTBot and you sit out of GPT training. Block OAI-SearchBot and you disappear from ChatGPT Search citations. Same pattern for ClaudeBot vs Claude-User vs Claude-SearchBot, and Applebot vs Applebot-Extended. Every pair needs its own explicit entry:

Apple updates its Applebot documentation explaining what it means to block Applebot-Extended vs Applebot https://t.co/IUUUslM49j https://t.co/7tFeBLPzvf

Barry Schwartz@rustybrickApr 28, 2025

Barry Schwartz at Search Engine Roundtable flagged the Apple clarification when it landed. Applebot handles Siri, Spotlight, and Safari suggestions; Applebot-Extended is the training opt-out for Apple Intelligence. Disallow: / on the parent Applebot cuts you from Siri; Disallow: / on Applebot-Extended does nothing to Siri visibility.

The First-Match Trap: How Group Precedence Works

Precedence in RFC 9309 looks simple until it collides with a CDN. Every AI crawler applies these rules as written:

1. Named group wins over the wildcard. If a file has both User-agent: GPTBot and User-agent: *, GPTBot uses only the rules in its named group.
2. Longest matching path wins inside a group. Allow: /blog/press-kit/ beats Disallow: /blog/ for URLs under press-kit.
3. Allow wins on a tie. Equal path length, Allow takes effect.
4. Multiple User-agent lines before the first rule form a shared group. How you apply one block to several crawlers.
5. Groups do not inherit from each other. Each group starts empty.

The trap sits in rule one. When Cloudflare's Managed robots.txt was rolled out, it prepended AI-blocking directives above whatever the customer wrote, and the first matching group wins:

A Turkish SEO practitioner debugging why client sites had gone missing from ChatGPT and Claude answers traced it to that behavior: the Cloudflare block sat above the user's rules, and the spec's first-match rule ignored every hand-written Allow directive below. What you write in your CMS is not necessarily what bots receive. Curl the live URL and confirm.

RANKCONTROL

We'll show you exactly where your brand stands in AI search.

No commitment. No credit card. See how ChatGPT, Perplexity, Claude, and Gemini talk about your brand today.

Get your free report→

Wildcards, Anchors, and Path Patterns

Path patterns have their own small syntax. Two characters do the work.

Two gotchas. Trailing wildcards are redundant: /api/* equals /api/. And the $ anchor is the only way to say "this exact URL." Without it, Disallow: /page also blocks /page-2, /page/subpage, /pages/, and anything else starting with those characters.

Escape Hatches: Allow, noai, and X-Robots-Tag

When you want to Disallow most of a directory but permit a slice, the Allow directive handles it. Because longer paths win, a longer Allow inside a shorter Disallow carves a legal exception:

User-agent: GPTBot
Disallow: /blog/
Allow: /blog/press-kit/

For HTML pages, the robots meta tag adds a second layer. noai and noimageai signal "do not use in AI training or AI image generation." Support is voluntary; OpenAI, Anthropic, and Adobe honor them for images, and many others do not. Treat these as supplements to robots.txt, not replacements.

The X-Robots-Tag HTTP response header carries the same directives server-side, which matters for non-HTML content. PDFs, feeds, and binary files cannot host a <meta> tag, so the header is the only escape route: X-Robots-Tag: noai, noimageai.

One critical distinction: Disallow and noindex are not the same thing. Disallow blocks the fetch entirely, so the crawler never sees the page. noindex needs the crawler to fetch the page first, then honor the tag by keeping it out of the index. If you Disallow a URL, the crawler never reaches your noindex tag, so the URL can still surface in search as a title-only link. Use Disallow for content crawlers should never touch. Use noindex for content you want crawled but not indexed.

Where robots.txt Alone Falls Short

The spec is compliant with itself. The world is not.

Perplexity was caught in August 2025 crawling sites that had blocked its bots via robots.txt and WAF rules. Cloudflare traced the traffic to Perplexity IPs rotating through browser-shaped user-agent strings, delisted the company as a Verified Bot, and Reddit and The New York Times filed federal suits later that year alleging DMCA anti-circumvention.

Cloudflare's own posture shifted with the September 15, 2026 policy change. Under the new default, Cloudflare classifies AI crawlers into three behaviors: Search, Agent, and Training. New domains have Training and Agent blocked by default on ad-monetized pages, regardless of the site's robots.txt.

Cloudflare just reshaped how AI crawlers get classified — and SEO teams need to pay attention before Sept 15. On its second "Content Independence Day," Cloudflare moved away from the blunt "block all AI bots" model toward a behaviour-based taxonomy. Instead of asking "is this

Pedro Dias@pedrodiasJul 2, 2026

Independent analyst Pedro Dias put the shift plainly. The infrastructure has moved from "is this an AI bot?" toward classifying by behavior. A User-agent directive names identities, not intents.

xAI's Grok publishes no bot token that shows up in HTTP requests, so robots.txt is useless against it. ai.txt and llms.txt are adjacent formats and neither is a blocking file.

RANKCONTROL

Know exactly what AI says about your competitors.

RankControl's Recon Agent monitors competitor citations across ChatGPT, Perplexity, Claude, and Gemini. See where they show up and you don't.

See how it works→

A Sensible SaaS robots.txt for 2026

A reasonable default posture for a SaaS site that wants to appear in AI citations, stay out of training pipelines it does not benefit from, and protect admin surfaces. Read the comments before pasting.

# robots.txt — SaaS defaults, 2026

# Protected paths
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/

# OpenAI: allow search/user, block training
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /

# Anthropic: allow search/user, block training
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /

# Google: allow Search, block Gemini training
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /

# Perplexity: allow, pair with WAF
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /

# Apple: allow Siri, block Apple Intelligence training
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /

# Bing/Copilot, Mistral, DuckAssist, You, Exa
User-agent: Bingbot
Allow: /
User-agent: MistralAI-User
Allow: /
User-agent: MistralAI-Index
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: YouBot
Allow: /
User-agent: Exa-Bot
Allow: /

# Meta: block training crawler, allow user fetches
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Allow: /

# Training-only bots: block
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Amazonbot
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Flip training vs search however you want. If you sell subscriptions and every citation matters, the shape above is defensible. If your content is proprietary research you never want reused, block everything except Search bots and pair it with CDN-layer WAF rules against spoofing.

Common Syntax Mistakes That Break Everything

Most broken robots.txt files fail in a small set of ways:

• User-agent: Claude-Bot (wrong: the token is ClaudeBot, one word).
• User-agent: ClaudeUser (wrong: the token is Claude-User, hyphenated).
• User-agent: GPT* (undefined: wildcards inside a User-agent token are invalid).
• Empty group: User-agent: Google-Extended followed by no Allow or Disallow does nothing.
• Treating Google-Extended like a crawler UA in log analysis (it never appears in logs; Googlebot does the fetch).
• Assuming line order changes precedence (it does not: longest path wins regardless of position).

The quiet failure is the file that reads correctly in the repo but is overwritten upstream. A GEO analyst reviewing Cloudflare data noted Anthropic's crawl-to-referral ratio runs around 100,000 to 1:

Blocking a training-only crawler that sends you nothing costs nothing. Blocking a citation crawler that sends you nothing costs nothing in referral traffic and everything in whether your brand exists inside Claude's answer.

You can maintain this by hand every quarter and diff the file against a curl of your live URL. Or RankControl flags it automatically the first time a bot you meant to allow starts hitting a Disallow.

RANKCONTROL

15+ content types. Published on your domain. Matched to your brand.

Guides, comparisons, listicles, case studies, and more. RankControl generates content that gets cited by ChatGPT, Claude, Perplexity, and more.

Start free→

The spec gets more powerful the more precisely you read it, and less useful the more you rely on it alone. Publishers who write the file carefully and verify what bots actually receive keep the levers robots.txt does control. The rest keep asking why their content is invisible to Claude, when the answer sits in a five-line block prepended above the directives they hand-wrote.