For two years, the usual way to answer "how many AI bots are crawling my site right now" was to SSH into a server and grep your Nginx logs. Ahrefs turned that grep into a dashboard on April 10, 2026, when it shipped Bot Analytics. It comes with four reports and a filter that isolates AI-only traffic, with a Cloudflare integration doing the ingestion. During the beta it's free on every paid Ahrefs plan.
Three months and a couple of meaningful updates later, it deserves a careful read, mostly for two things: a single Ahrefs stat that should change how SaaS founders think about bot traffic, and a limitation that quietly rules out half the internet.
What Ahrefs actually built
The setup is simple. Bot Analytics pulls HTTP request data straight out of Cloudflare, through a Worker on any Cloudflare plan or through Logpush if you're on Cloudflare Enterprise, and spreads it across four connected reports:
| Report | What it shows |
|---|---|
| Overview | One line chart of all bot activity, absolute or relative |
| Bots | Traffic per crawler, including Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bingbot, ChatGPT-User, Claude-User, OAI-SearchBot, Google-Extended, Bytespider, CCBot, Meta-ExternalAgent, Amazonbot and Ahrefs' own AhrefsBot |
| Categories | The same traffic grouped into 12 buckets covering search engine, AI crawler, AI assistant, SEO tool, social platform, monitoring, advertising, security and a handful of others |
| Crawled Pages | The specific URLs bots hit, with visit counts per bot per URL |
Flip the "AI bots" toggle and all four reports narrow to AI traffic only.
Two updates since launch matter a lot. On May 8 HTTP status codes came into the view, so you can see where bots run into 404s or get stuck in redirect loops. Bot verification followed on June 26. It checks incoming requests against Cloudflare's verified-bots list and known IP ranges, and traffic pretending to be Googlebot gets a bucket of its own. This was Ahrefs' post at launch:
NEW: Bot Analytics 🤖 (free while in beta!) Monitor which bots (such as AI bots) visit your site and how often. Optimize your crawl budget efficiently. Works with a Cloudflare integration on any plan https://t.co/Wcv07CMVG4
Ahrefs@ahrefsApr 10, 2026
You can view anything from hourly to monthly, pick custom dates and export to CSV or Google Sheets. There's no dedicated API endpoint yet, though, which will bother you if you'd planned to pipe this into your own dashboards.
26 content formats. Published on your domain. Matched to your brand.
Guides, comparisons, listicles, case studies, and more. RankControl generates content that gets cited by ChatGPT, Perplexity, Claude, Gemini, Grok, and Google AI Mode.
The stat that changes founder behavior
For years, bot analytics was a story about bandwidth bills. AI bots hammer the site until Nginx runs hot, and then somebody opens a Vercel invoice nobody budgeted for. Ahrefs' May 15 study turned it into a revenue story. AI search visitors made up 0.5% of visitors and 12.1% of signups, which works out to a conversion rate 23 times higher than organic search.
After a number like that, you stop asking "how do I stop these bots eating my bandwidth" and start asking "how do I tell whether the ones that matter are getting through." The second question is harder than it sounds, because "AI bots" covers two groups with little in common.
Training crawlers such as GPTBot and ClaudeBot are there to feed the next model release, and none of their visits will ever send you a customer. Retrieval agents work differently. ChatGPT-User and Claude-User, along with OAI-SearchBot, turn up when a real person asks the assistant a real question, and their visits are the ones directly tied to a citation landing in the answer. Splitting them apart in the Bots report is, for a SaaS funnel, the first properly useful thing the tool does.
The crawl-to-referral gap is the diagnosis
With the AI filter on, the number I'd stare at is each bot's ratio of crawls to referrals. Cloudflare's May 2026 data via WorkOS shows how wide the spread gets:
| Bot | Crawls per referral |
|---|---|
| ClaudeBot | ~23,951:1 in Q1 2026, improving to ~11,122:1 by late May |
| GPTBot | Roughly 1,255:1 |
| PerplexityBot | 111:1 |
| Googlebot | 4.9:1 |
A huge gap means the relationship is training-only. The bot reads your content to improve its next model, and none of those reads come back as citations that send people to you. When the gap starts closing, particularly for retrieval agents like Perplexity's and OAI-SearchBot, you're seeing what real AI search visibility looks like from the crawler side. Say Bot Analytics shows Claude-User visiting your top ten pages twice a week: that's visibility in Claude answers as it shows up in your server logs.
The loop I'd build checks that crawl pattern against citation data from a monitoring tool. Plenty of crawling with no citations points to a problem with how your content is structured. Citations with little crawl volume behind them are a source-attribution puzzle worth digging into.
People on r/webdev have been staring at this problem for a year without the vocabulary for it:
ClaudeBot is hammering my server with almost a million requests in one day
Just checked my crawler logs for the last 24 hours and ClaudeBot (Anthropic) hit my site ~881,000 times. That’s basically my entire traffic for the day. I don’t mind legit crawlers like Googlebot/Bingbot since they at least help with indexi...
The top-voted reply is one line long: "Cloudflare has a setting to block AI scrapers." The comment that matters more sits further down, and it's a question: "how did you get this report, which tool is it?" The thread runs two thousand upvotes deep, full of people who could see the traffic and had no way to measure the intent behind it. Bot Analytics is Ahrefs answering that commenter.
Your competitors are getting cited by AI. You're not.
Every day without citation tracking is a day your competitors pull ahead in ChatGPT, Perplexity, and Claude.
The caveat half the internet should know
Bot Analytics only works right now if Cloudflare sits in front of your site. The Worker method runs on any Cloudflare plan, free included, and Logpush runs on Enterprise. If your SaaS is hosted directly on Vercel or AWS CloudFront, or on Fastly or Netlify, or on a bare VPS without Cloudflare, Bot Analytics has no data source to read. Ahrefs' developer docs say "more data sources coming soon," and the Canny board has active requests, but as of the beta nobody has an ETA.
There are two smaller problems as well. The tool doesn't confirm attribution: a bot crawling your page is a signal, and it proves nothing about whether that page made it into an answer. To close the loop you'll still need GA4 referrers (chatgpt.com, claude.ai, perplexity.ai) plus a citation tracker. The other problem is cost, since Meta's crawler alone can chew through nine hundred gigabytes of bandwidth in a month:
Meta's AI crawler scraped my site 7.9 million times in 30 days. 900+ GB of bandwidth and massive server logs before I noticed, cool cool cool.
Bot Analytics shows you that traffic, and deciding what to do about it is still on you. Robots.txt only works when a crawler chooses to honor it, which most training crawlers do and some don't, so the real blocking happens through rate limits at the CDN layer. And if Cloudflare's September 15 default-blocking policy takes effect before you've made up your mind, the default makes the call for you.
What to actually do this week
If you're on Cloudflare already, install the Worker (about five minutes for a Standard site) and give it a week to collect data. Then I'd work through it in this order.
Turn on the AI filter and separate training crawlers from retrieval agents in the Bots report, since the retrieval visits are the ones tied to signups. Next, sort the Crawled Pages report by AI bot visits. Whatever sits at the top is what AI models treat as your most relevant material, and those are the pages to keep fresh.
After that, check the HTTP status codes. A 404 or a redirect loop in an AI-bot row is quietly costing you citations you'd otherwise earn, and fixing it is often an hour's work. Look at the bot verification split too. Anything in the "spoofed" bucket is bandwidth spent for zero visibility upside, so I'd rate-limit it or block it at the CDN.
Finish by checking all of it against citation tracking data, because that's how you learn whether the bot activity turns into answers and whether those answers bring in customers.
Setup takes under an hour, and the monthly review under fifteen minutes. Or you can have agents on our side read the citation half of that loop every week, so the crawl data has something to be compared against. That's the closed loop the two tools were meant to give you together.
200+ SaaS teams already track their AI citations.
They know exactly when ChatGPT mentions their brand, and when it stops. Do you?
Nobody else has made AI crawler behavior this readable without asking you to build a log-parsing pipeline of your own. The Cloudflare gating is what that shortcut costs. For a SaaS already sitting behind Cloudflare, Bot Analytics is the observability layer the AI referral era has been missing, and I'd install it before the September policy changes start moving the crawl numbers anyway.
