---
title: "AI crawlers"
description: "Which AI crawlers to allow on your site, which you can block, and how to set it in Cloudflare without losing citations or Google traffic."
---

> Documentation Index
> Fetch the complete documentation index at: https://rctrl.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawlers

AI engines read your site with different crawlers for different jobs. Some
build the search indexes that decide whether your pages appear in AI
answers. Some fetch a page live when a user asks about it. Some only
collect training data. What you allow decides whether AI answers can cite
you.

## You do not need to check this yourself

We test your site every day, pretending to be each AI crawler. If any of
them are blocked, we tell you: a warning appears on your Analytics screen
and you get a notification. There is nothing for you to run or install.

This matters because a block is impossible to spot on your own. Your site
opens normally in your browser while quietly turning the crawlers away, so
everything looks fine. Only a test that arrives as a crawler can see it.

If we do flag it, the fix is a setting at your hosting or security
provider, not a change to your website. The most common cause is a
Cloudflare setting called **Block AI bots**, which is often switched on
without anyone choosing it.

If someone else looks after your website, send them the section below.

## Always allow these

These crawlers are how your pages get into AI answers. Blocking any of
them removes you from that engine's results.

| Crawler | Who | What it does |
|---|---|---|
| `OAI-SearchBot` | OpenAI | Builds the ChatGPT search index. Sites that block it do not appear in ChatGPT search answers. |
| `ChatGPT-User` | OpenAI | Fetches your page live when a ChatGPT user asks about it |
| `Claude-SearchBot` | Anthropic | Indexes for Claude's web search |
| `Claude-User` | Anthropic | Fetches your page live when a Claude user asks |
| `PerplexityBot` | Perplexity | The index Perplexity cites from. Perplexity has no separate training crawler. |
| `Perplexity-User` | Perplexity | Live fetches for Perplexity answers |
| `Googlebot` | Google | Google Search, AI Overviews, and AI Mode all ride the same index |
| `Bingbot` | Microsoft | Bing and Copilot answers |

## Training crawlers: your choice

| Crawler | Who | What it does |
|---|---|---|
| `GPTBot` | OpenAI | Collects training data for OpenAI models. Blocking it does not affect ChatGPT search. |
| `ClaudeBot` | Anthropic | Collects training data for Anthropic models. Blocking it does not affect Claude's web search. |
| `Google-Extended` | Google | A robots.txt token, not a crawler. It covers Gemini training AND grounding, so blocking it can also cost you Gemini citations. We recommend leaving it open. |

Our recommendation: leave training crawlers open while you are building
your name. Models learn brands from the whole web, and while your brand is
young, your own site is the main place they can learn yours from. Once your
brand lives on more sites than your own, blocking training is a reasonable
choice, and it costs no citations.

To block training only, add this to robots.txt and change nothing else:

```txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
```

## Cloudflare settings

For whoever manages the site. Check these even if nobody changed them,
because **Block AI bots is switched on by default on many Cloudflare
accounts** and it blocks the search crawlers, not only the training ones.
Cloudflare's own AI Crawl Control screen will show the blocked requests,
which never appear in normal website analytics.

Go to **Security**, then **Settings**.

- **Block AI bots** blocks the search crawlers above too, not just
  training. Set its scope to **Do not block (allow crawlers)** and manage
  training through robots.txt instead.
- In its September 15 choice about mixed-purpose crawlers, pick **"Mixed
  purpose crawlers will continue to be allowed."** Cloudflare counts
  Googlebot, Bingbot, and Applebot as mixed-purpose. The other choice
  blocks them at the network level and removes you from Google, including
  AI Overviews.
- **AI Crawl Control**, its own section in the Cloudflare sidebar, sets
  crawlers by category. Keep **Search** and **Agent** on allow. Set
  **Training** however you decided above, but prefer robots.txt for
  training blocks: the Cloudflare Training category also catches the
  mixed-purpose crawlers.
- **Managed robots.txt**, on the same AI Crawl Control screen, lets
  Cloudflare write content signals into your robots.txt for you. It adds a
  `# BEGIN Cloudflare Managed content` block with
  `Content-Signal: search=yes, ai-train=no, use=reference`. These are
  requests, not blocks, so they cost you no citations. Turn it off if you
  would rather own that file yourself. If you find signals in your
  robots.txt that you never wrote, this is where they came from.
- **Bot fight mode** can stay on. It challenges scripted traffic and does
  not block the verified crawlers above.
- Every crawler above is on Cloudflare's verified list, so keeping
  verified bots allowed covers them.

## How we check this

RankControl tests your site every day as the AI search crawlers. If your
firewall or robots.txt blocks them, the Analytics screen shows a
diagnostic and you get a notification. Blocking the training crawlers
never triggers an alert.

Allowing RankControl's own crawler is a separate, single-IP rule: see
[RankControl crawler](/docs/bot).

Source: https://rctrl.com/ai-crawlers/index.mdx