You open GA4 and there's no sign of ChatGPT anywhere. Then someone on your team opens the Cloudflare dashboard and finds that OpenAI's crawler requested your pages tens of thousands of times last month. Both views are correct. They're measuring different visitors.
Those requests are AI bot visits: software from an AI company fetching your pages, sometimes to train a model, sometimes to answer a person's question. Here's what the different kinds are, why your analytics never shows them, what they can and can't tell you, and how much tracking they're worth.
Four kinds of AI bot visits
AI companies now send different bots for different jobs, and each one means something different for you.
| Kind | What it does | Examples | Does robots.txt stop it? |
|---|---|---|---|
| Training crawler | Collects pages that may train future models | GPTBot, ClaudeBot | Yes, if you disallow it |
| Search crawler | Builds the index AI answers draw from | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Yes |
| User-triggered fetch | Opens a page because someone just asked about it | ChatGPT-User, Claude-User, Perplexity-User | Often not |
| Agent | Browses and takes actions for a person | Google-Agent | No, it's user-triggered |
The fetch column is where vendors differ. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic's docs say Claude-User does respect robots.txt.
The agent row is the newest. Google documents a separate user agent, Google-Agent, that's "used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request."
OpenAI takes a different approach. Its crawler docs list no separate agent user agent, so when ChatGPT browses on someone's behalf, the visit shows up as ChatGPT-User. You can't tell a quick lookup from an agent session by name alone.
Why GA4 never shows them
GA4 counts a visit when its JavaScript tag runs in a browser. AI crawlers mostly don't run JavaScript. Vercel and MERJ studied crawler traffic across Vercel's network in late 2024 and found that "none of the major AI crawlers currently render JavaScript."
The crawlers do download script files sometimes. The same study found ChatGPT's crawler fetched JavaScript files in about 12% of requests and Claude's in about 24%, but neither ran them. A page that never runs your tag never becomes a GA4 session.
Results consistently show that none of the major AI crawlers currently render JavaScript. However, Google's Gemini and AppleBot enable full JavaScript rendering, which may push others to follow.
Vercel@vercelDec 17, 2024So AI bot visits live in a different place from your normal analytics: server logs, your CDN dashboard, and any analytics tool that reads requests on the server rather than in the browser.
How much crawling turns into visitors
Most AI bot traffic takes far more than it sends back. Cloudflare compares how many pages each AI company crawls with how many visitors that company's product sends to sites. In early August 2025, its numbers were about 50,000 crawls per referred visit for Anthropic, roughly 887 for OpenAI and around 118 for Perplexity.
The same analysis found that training accounted for about 80% of AI crawling, which is why crawl volume and referral traffic can move in opposite directions. Two caveats come with these ratios. They're from 2025, and Cloudflare notes that Claude's app sends no referrer header, which makes its ratio look more one-sided than it is.
Which bots do most of the crawling? Among AI crawlers alone, Cloudflare's May 2025 data put GPTBot at 30% of requests, ahead of ClaudeBot at 21% and Meta's crawler at 19%.
AI bots generally crawl for training, search indexing, or in response to a user action. @CloudflareRadar now provides a "crawl purpose" perspective on AI bot traffic trends -- see more at https://t.co/NpMatAcc8N https://t.co/9z6M6YxOGa
Cloudflare Radar@CloudflareRadarAug 5, 2025
The takeaway for a marketer is simple. A big crawl number mostly means your pages are being collected. It says very little about whether anyone is reading your brand's name in an answer.
AI search traffic grew 835% this year. Is your content ready?
RankControl generates 26 content formats optimized for ChatGPT, Claude, and Perplexity. Published on your domain, matched to your brand.
Demand bots are the ones worth watching
Of the four kinds, user-triggered fetches are the most interesting to a marketer. A visit from ChatGPT-User or Perplexity-User usually means a person asked something a few seconds ago and the engine decided your page might answer it. We call these demand bots, because they're the closest thing to a signal of real interest that bot traffic gives you.
Most sites don't treat them that way yet. On September 27, we checked the robots.txt files of ten large sites, from The New York Times and GitHub to Forbes and HubSpot. Here's how many named each AI bot:
| Bot | Sites naming it |
|---|---|
| GPTBot (training) | 6 of 10 |
| ChatGPT-User (demand) | 4 of 10 |
| ClaudeBot (training) | 6 of 10 |
| Claude-User (demand) | 4 of 10 |
| PerplexityBot (search) | 5 of 10 |
| Perplexity-User (demand) | 3 of 10 |
| Google-Agent (agent) | 0 of 10 |
Sites write rules for training crawlers more often than for the fetchers that follow real questions, and none had a rule for Google's agent. That isn't wrong, since blocking training is a common choice, but it shows where attention has gone. The bot that signals demand is the one getting the least thought.
Where you can see AI bot visits
You have three places to look, depending on your setup:
- Your CDN dashboard. Cloudflare and similar services show bot traffic by user agent, and some split it by AI purpose.
- Your hosting or server logs. Every request lands here, including ones your CDN didn't block.
- Analytics tools that count requests on the server side, rather than relying on a browser tag.
If you want to go deeper, our guide to auditing AI crawlers in server logs walks through the log work step by step.
This also matters for the numbers you already report. A developer in r/webdev asked in August how much of what analytics calls direct traffic is even human anymore, since bots counted as visits drag down conversion rates. Many replies came from people facing the same problem.
How are you separating actual users from AI/bot traffic now?
With AI crawlers and agents hitting websites constantly, how much of what analytics calls “direct traffic” is even human anymore? If bots are getting counted as visits, conversion rates automatically look worse. And then decisions about lan...
Telling a real bot from an impostor
A user agent is just a label, and anyone can copy it. Google's own list of fetchers carries a warning that "the user agent string can be spoofed," and it points site owners to IP-range and DNS checks instead. A spike from "GPTBot" in your logs may be a scraper wearing OpenAI's name.
Agents make this harder, because they browse like people. Cloudflare proposed a fix in May 2025 called Web Bot Auth, where a bot signs each request so a site can check who sent it. OpenAI signed its Operator agent's traffic this way at launch, and Google's fetcher page notes that Google is experimenting with the same protocol. Until more vendors adopt it, treat any single bot name in your logs as a claim, not a fact.
What bot visits tell you, and what they don't
| They can tell you | They can't tell you |
|---|---|
| Which pages AI crawlers fetch, and how often | Whether an AI answer cited you |
| Whether a bot is being blocked or getting errors | What question the person asked |
| Whether bots came back after you updated a page | Whether a human ever clicked through |
| Which AI companies are most interested in your content | How your brand was described |
The left column is useful for spotting problems. A CDN setting that starts returning errors to OAI-SearchBot, for example, would show up here long before you noticed missing citations. Our Cloudflare Bot Fight Mode checklist covers the settings that cause that most often.
The right column needs a different tool. To know whether you're cited, you have to check the AI answers themselves, which is what citation tracking is for.
Built by the team that got cited in 48 hours.
Content generation, backlink building, AI visibility tracking, and Google rankings. One platform, zero guesswork.
So should you track them?
Yes, but lightly. Here's what we'd do:
- Look once a month at which AI bots reached your most important pages.
- Check that none of the search crawlers or demand bots are being blocked or getting errors.
- Glance again after any change to robots.txt, your CDN or your site structure.
- Don't report raw crawl volume as a performance metric.
It deserves more attention in two cases. Large content sites can see real bandwidth costs from training crawlers, and anyone who just changed firewall or bot settings should confirm nothing important was cut off. Our complete AI crawler list has every user agent string, and our post on whether to block AI crawlers covers the blocking decision.
Budget 20 minutes a month for the check once you know where to look.
Bot visits and citations in one place
You can keep this manual: a monthly look at your CDN, a note of anything blocked, and a separate check of AI answers for your top buyer questions.
Or RankControl can put both halves together. Our analytics count AI crawler visits and AI-referred visits alongside your normal traffic, and our AI visibility tracking checks up to 50 buyer questions every week across seven AI engines, so you can see whether the bots that visit are turning into citations.
See your first AI citation report in under 5 minutes.
No setup calls. No onboarding meetings. Connect your domain and see where AI mentions your brand right now.
