AI engines read your site with different crawlers for different jobs. Some build the search indexes that decide whether your pages appear in AI answers. Some fetch a page live when a user asks about it. Some only collect training data. What you allow decides whether AI answers can cite you.
You do not need to check this yourself
We test your site every day, pretending to be each AI crawler. If any of them are blocked, we tell you: a warning appears on your Analytics screen and you get a notification. There is nothing for you to run or install.
This matters because a block is impossible to spot on your own. Your site opens normally in your browser while quietly turning the crawlers away, so everything looks fine. Only a test that arrives as a crawler can see it.
If we do flag it, the fix is a setting at your hosting or security provider, not a change to your website. The most common cause is a Cloudflare setting called Block AI bots, which is often switched on without anyone choosing it.
If someone else looks after your website, send them the section below.
Always allow these
These crawlers are how your pages get into AI answers. Blocking any of them removes you from that engine’s results.
| Crawler | Who | What it does |
|---|---|---|
OAI-SearchBot |
OpenAI | Builds the ChatGPT search index. Sites that block it do not appear in ChatGPT search answers. |
ChatGPT-User |
OpenAI | Fetches your page live when a ChatGPT user asks about it |
Claude-SearchBot |
Anthropic | Indexes for Claude’s web search |
Claude-User |
Anthropic | Fetches your page live when a Claude user asks |
PerplexityBot |
Perplexity | The index Perplexity cites from. Perplexity has no separate training crawler. |
Perplexity-User |
Perplexity | Live fetches for Perplexity answers |
Googlebot |
Google Search, AI Overviews, and AI Mode all ride the same index | |
Bingbot |
Microsoft | Bing and Copilot answers |
Training crawlers: your choice
| Crawler | Who | What it does |
|---|---|---|
GPTBot |
OpenAI | Collects training data for OpenAI models. Blocking it does not affect ChatGPT search. |
ClaudeBot |
Anthropic | Collects training data for Anthropic models. Blocking it does not affect Claude’s web search. |
Google-Extended |
A robots.txt token, not a crawler. It covers Gemini training AND grounding, so blocking it can also cost you Gemini citations. We recommend leaving it open. |
Our recommendation: leave training crawlers open while you are building your name. Models learn brands from the whole web, and while your brand is young, your own site is the main place they can learn yours from. Once your brand lives on more sites than your own, blocking training is a reasonable choice, and it costs no citations.
To block training only, add this to robots.txt and change nothing else:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Cloudflare settings
For whoever manages the site. Check these even if nobody changed them, because Block AI bots is switched on by default on many Cloudflare accounts and it blocks the search crawlers, not only the training ones. Cloudflare’s own AI Crawl Control screen will show the blocked requests, which never appear in normal website analytics.
Go to Security, then Settings.
- Block AI bots blocks the search crawlers above too, not just training. Set its scope to Do not block (allow crawlers) and manage training through robots.txt instead.
- In its September 15 choice about mixed-purpose crawlers, pick “Mixed purpose crawlers will continue to be allowed.” Cloudflare counts Googlebot, Bingbot, and Applebot as mixed-purpose. The other choice blocks them at the network level and removes you from Google, including AI Overviews.
- AI Crawl Control, its own section in the Cloudflare sidebar, sets crawlers by category. Keep Search and Agent on allow. Set Training however you decided above, but prefer robots.txt for training blocks: the Cloudflare Training category also catches the mixed-purpose crawlers.
- Managed robots.txt, on the same AI Crawl Control screen, lets
Cloudflare write content signals into your robots.txt for you. It adds a
# BEGIN Cloudflare Managed contentblock withContent-Signal: search=yes, ai-train=no, use=reference. These are requests, not blocks, so they cost you no citations. Turn it off if you would rather own that file yourself. If you find signals in your robots.txt that you never wrote, this is where they came from. - Bot fight mode can stay on. It challenges scripted traffic and does not block the verified crawlers above.
- Every crawler above is on Cloudflare’s verified list, so keeping verified bots allowed covers them.
How we check this
RankControl tests your site every day as the AI search crawlers. If your firewall or robots.txt blocks them, the Analytics screen shows a diagnostic and you get a notification. Blocking the training crawlers never triggers an alert.
Allowing RankControl’s own crawler is a separate, single-IP rule: see RankControl crawler.