Somewhere this week, a site owner pastes one new line into robots.txt and feels good about blocking AI training. They won't notice they also opened a different door. A reader on r/Agentic_SEO raised that exact worry the day after Google and Bing announced a shared opt-out. It's worth sorting out before you touch your own file.
What Google and Bing actually announced on October 30
Google's own blog laid it out on October 30. Instead of writing separate rules for every AI developer, site owners can now use one shared token in robots.txt. It sets a broad choice across every AI provider that agrees to honor it. Bing posted almost the same news on the same day.
Anthropic's help center confirmed days later that Claude had already started reading the new signal. TechRadar reported that OpenAI backs it too.
The token comes out of an Internet-Draft written by the IETF's AIPREF working group, the same group trying to build one shared way for the web to state AI preferences. Google has offered its own training opt-out, Google-Extended, since 2023. This new token sits beside it rather than replacing it.
The token, in plain robots.txt
Here's what it looks like in practice. The exact string matters. It's two underscores, then usage_ai-training, straight from the IETF draft and matched in both Google's and Bing's own examples. The draft says consumers must match it exactly, aside from capitalization, the same way they'd match any other bot name in the file.
# Opt out of AI training (shared token)
User-Agent: __usage_ai-training
Disallow: /
# Still allow Google's AI Overviews and Gemini grounding
User-Agent: Google-Extended
Allow: /
Drop the first block into your root robots.txt. Once a given AI company starts honoring the token, your site opts out of training with all of them at once. No more hunting down GPTBot, ClaudeBot and a dozen other names one at a time. Want to see what the shared token replaces? The full list of individual tokens is in our robots.txt directive reference.
The catch nobody's talking about
This is the part that should slow you down. Google's post says something specific here. If a site uses the shared token to opt out of training and doesn't separately name Google-Extended, Google reads that as disallowing Google-Extended too. Only sites that explicitly allow Google-Extended stay eligible to appear and get clicks through grounding in Gemini apps.
One reply on that r/Agentic_SEO thread made a sharp point about where this goes wrong. Most people won't write that line by hand.
They'll pull it from a plugin default or a copy-paste template, read "blocks AI training," and never check what else it quietly switches off. A year from now, some of those sites will be confused about where their AI answer traffic went.
Google announced AI training opt-out in robots.txt
Time to decide which matters more for your site: control over AI training or visibility in AI answers. Google plans to support a shared AI training opt-out in robots.txt in 2027. Currently, a site that wants to opt out of AI training has to...
26 content formats. Published on your domain. Matched to your brand.
Guides, comparisons, listicles, case studies, and more. RankControl generates content that gets cited by ChatGPT, Perplexity, Claude, Gemini, Grok, Google AI Mode, and Google AI Overviews.
What Bing adds on top
Bing's version goes a step further than Google's. Microsoft is retiring its old NoArchive and NoCache tags, replacing them with two separate settings instead of one. Training is its own choice now, and grounding, meaning whether your content can inform a live AI answer, is a separate one. If your site currently relies on either of the old tags, Bing has set a hard deadline for switching to the new settings.
Marie Haynes, who covers Google's own search-quality changes closely, flagged the shared token the day after Google's post went up.
For those who don't want AI training on your content, there will soon be a new robots.txt control that instructs all AI tools not to train on your content: _usage_ai-training. This means you don't have to block bots individually. Google says, "Instead of managing separate https://t.co/WW0LGHlZSD
Marie Haynes@Marie_HaynesOct 31, 2026
What to actually set today, in three situations
Most sites fall into one of three camps, and the right move depends on which one you're in.
- Fine with training? Do nothing yet. Allow is the default when no preference is published, for every vendor involved so far.
- Want citations, not training? Use the shared token to disallow training. Then add an explicit allow for Google-Extended and any other vendor token you care about, the way the example above does.
- Want neither? Disallow the shared token and skip the vendor-specific allows entirely.
Before you change anything, it helps to know what your site currently allows. Our free AI crawler checker reads your live robots.txt against the crawlers that matter and shows your starting point in about 20 seconds.
Your competitors are getting cited by AI. You're not.
Every day without citation tracking is a day your competitors pull ahead in ChatGPT, Perplexity, and Claude.
This only works if everyone keeps their word
None of this is enforced by a browser or a court. Robots.txt has always run on trust. A few people on r/TechSEO were quick to point out that telling a crawler not to do something is closer to a sign on a door than a lock on it.
The companies behind this signal have a public reason to keep their word now that they've said so on their own blogs. But the underlying document is still an early-stage IETF draft, not a finished standard.
The four companies aren't moving at the same speed either. Anthropic's help center says it started honoring the signal on October 30 and is rolling it out across its services over time. Bing says the split preferences start counting from February 1, 2027. Google's post commits only to "2027," with no firmer date attached yet.
The IETF group isn't finished either. A companion draft, still being revised as of September 2026, covers how to attach the same preferences through HTTP headers too, alongside robots.txt. The channel you use to express a preference is likely to grow past one text file over the next year, and the vocabulary behind both drafts is still open to change as more companies weigh in.
The manual way, and the easier one
You can do this by hand. Read each vendor's crawler docs, get the token right, decide which bots to let back in, then check again whenever another company signs onto the draft.
Or let RankControl's weekly checks across all seven AI engines show you, in one dashboard, whether a citation actually disappeared, instead of guessing why traffic dropped three months from now. Tracking real AI search visibility matters more than any single robots.txt line. A token like this changes what a crawler is allowed to do. It doesn't change whether an AI engine picks you as its source, and that's the number worth watching.
200+ SaaS teams already track their AI citations.
They know exactly when ChatGPT mentions their brand, and when it stops. Do you?
