For the last two years, "how many AI bots are crawling my site right now" has been a question you answered by SSH-ing into a server and grep-ing Nginx logs. On April 10, 2026, Ahrefs shipped Bot Analytics and made that grep a dashboard. Free during beta on all paid Ahrefs plans, four reports, one filter for AI-only traffic, and a Cloudflare integration doing the ingestion. Three months in, with a couple of important feature updates layered on top, it's worth reading properly. Here's what the data actually shows, the one stat that changes the calculus for SaaS founders, and the caveat that quietly rules out half the internet.
What Ahrefs Actually Built
Bot Analytics ingests HTTP request data straight from Cloudflare, either via a Worker on any Cloudflare plan or via Logpush on the Cloudflare Enterprise tier, and drops it into four connected reports inside Ahrefs:
- Overview: a single line chart of all bot activity, absolute or relative.
- Bots: traffic broken down by individual crawler, including Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bingbot, ChatGPT-User, Claude-User, OAI-SearchBot, Google-Extended, Bytespider, CCBot, Meta-ExternalAgent, Amazonbot, and Ahrefs' own AhrefsBot.
- Categories: the same traffic grouped into 12 buckets covering search engine, AI crawler, AI assistant, SEO tool, social platform, monitoring, advertising, security, and a handful of others.
- Crawled Pages: the specific URLs bots hit, with visit counts per bot per URL.
A single "AI bots" toggle filters all four reports to AI-only traffic. The company shipped two important updates since launch. On May 8, HTTP status codes joined the view, so you can see where bots hit 404s or get looped in redirects. On June 26, bot verification landed, comparing incoming requests against Cloudflare's verified-bots list and known IP ranges so spoofed Googlebot traffic gets its own bucket. Announcement from Ahrefs at launch:
NEW: Bot Analytics 🤖 (free while in beta!) Monitor which bots (such as AI bots) visit your site and how often. Optimize your crawl budget efficiently. Works with a Cloudflare integration on any plan https://t.co/Wcv07CMVG4
Ahrefs@ahrefsApr 10, 2026
Data granularity runs from hourly to monthly with a custom date picker, and export is CSV or Google Sheets. There isn't a dedicated Bot Analytics API endpoint yet, which is worth knowing if you were planning to pipe the data into your own dashboards.
15+ content types. Published on your domain. Matched to your brand.
Guides, comparisons, listicles, case studies, and more. RankControl generates content that gets cited by ChatGPT, Claude, Perplexity, and more.
The Stat That Changes Founder Behavior
Bot analytics as a category has been a bandwidth-cost story for years. AI bots hammer a site, Nginx runs hot, someone gets a Vercel bill they didn't budget for. Ahrefs' own May 15 study reframed the entire conversation with one number: AI search visitors are 0.5% of visitors but 12.1% of signups, which is a 23x higher conversion rate than organic search. Suddenly the question isn't "how do I stop these bots eating my bandwidth" so much as "how do I tell whether the ones that matter are getting through."
That shift matters because the AI bot pool is not one thing. Training crawlers like GPTBot and ClaudeBot exist to feed the next model release; their visits will never send you a customer. Retrieval agents like ChatGPT-User, Claude-User, and OAI-SearchBot show up when a real user asks the assistant a real question, and their visits are the ones directly correlated with the citation that lands in the response. Bot Analytics lets you split those two groups apart in the Bots report, which for anyone building a SaaS funnel is the first useful thing this tool does.
The Crawl-to-Referral Gap Is the Diagnosis
Once the AI filter is on, the number worth staring at is the ratio between crawl volume and referral traffic, per bot. Cloudflare's May 2026 data via WorkOS tells the story cleanly:
- ClaudeBot: ~23,951 crawls per referral in Q1 2026, improving to ~11,122:1 by late May.
- GPTBot: roughly 1,255:1.
- PerplexityBot: 111:1.
- Googlebot: 4.9:1.
A large gap between crawls and referrals is a training-only relationship. The bot is reading your content to improve the next model version, and none of the reads are surfacing as citations that send users back. A closing gap, especially in the retrieval agents like Perplexity and OAI-SearchBot, is what actual AI search visibility looks like on the crawler side. If Bot Analytics shows Claude-User visiting your top ten pages twice a week, that is what earning visibility in Claude answers looks like in server logs.
Cross-referencing that pattern against citation data from a monitoring tool is the closed loop worth building. Crawl volume without citations is a content-structure problem. Citations without crawl volume is a source-attribution puzzle worth investigating.
The r/webdev community has spent a year staring at the same problem without the vocabulary to describe it:
ClaudeBot is hammering my server with almost a million requests in one day
Just checked my crawler logs for the last 24 hours and ClaudeBot (Anthropic) hit my site ~881,000 times. That’s basically my entire traffic for the day. I don’t mind legit crawlers like Googlebot/Bingbot since they at least help with indexi...
The top-voted response is one line: "Cloudflare has a setting to block AI scrapers." The most consequential comment further down is a question: "how did you get this report, which tool is it?" That thread ran two thousand upvotes deep on people who could see the traffic and couldn't measure the intent behind it. Bot Analytics is Ahrefs answering that question.
Your competitors are getting cited by AI. You're not.
Every day without citation tracking is a day your competitors pull ahead in ChatGPT, Perplexity, and Claude.
The Caveat Half the Internet Should Know
Bot Analytics is Cloudflare-only right now. The Worker method works on any Cloudflare plan including free, and Logpush works on Cloudflare Enterprise. If your SaaS is hosted directly on Vercel, AWS CloudFront, Fastly, Netlify, or a bare VPS without Cloudflare in front of it, Bot Analytics does not have a data source to read. Ahrefs' developer docs mention "more data sources coming soon," and the Canny board has active requests, but as of the beta there is no ETA.
Two related caveats worth flagging. The tool doesn't confirm attribution; a bot crawling your page is signal, not proof the page ended up in an answer. Cross-reference with GA4 referrers (chatgpt.com, claude.ai, perplexity.ai) and with a citation tracker to close the loop. And Meta's crawler alone can burn nine hundred gigabytes of bandwidth in a month:
Meta's AI crawler scraped my site 7.9 million times in 30 days. 900+ GB of bandwidth and massive server logs before I noticed, cool cool cool.
Bot Analytics gives you the visibility to make a decision about that traffic. It doesn't make the decision for you. Robots.txt is still a gentleman's agreement most training crawlers respect and some do not, and rate-limiting at the CDN layer is where the actual blocking happens. If Cloudflare's September 15 default-blocking policy hits before you've made the call, the default gets made for you.
What to Actually Do This Week
If you're on Cloudflare, install the Worker (roughly five minutes for a Standard site), let it collect for a week, then run this checklist:
- Filter to AI bots. Split training crawlers from retrieval agents in the Bots report. Retrieval visits are the ones tied to signups.
- Rank the Crawled Pages report by AI bot visits. Whichever URLs sit at the top are the ones AI models consider most relevant. Those are the priority pages for the content engine to keep fresh.
- Look at HTTP status codes. 404s and redirect loops in AI-bot rows are quietly costing you citations you would otherwise earn. Fixing them is often a one-hour job.
- Check the bot verification split. Any traffic in the "spoofed" bucket is bandwidth for zero visibility upside. Rate-limit or block at the CDN.
- Cross-reference with citation tracking data. That's how you tell whether the bot activity is generating the answers that generate the customers.
Total setup: under an hour, monthly review under fifteen minutes. Or agents on our side read the citation half of that loop continuously so crawl data has something to be compared against, which is the closed loop the two-tool combination was built to give you.
200+ SaaS teams already track their AI citations.
They know exactly when ChatGPT mentions their brand, and when it stops. Do you?
Bot Analytics is the closest anyone has come to making AI crawler behavior legible without a log-parsing pipeline of your own. The Cloudflare gating is the price for the shortcut. But for the Cloudflare-hosted SaaS, this is the observability layer the AI referral era was missing, and the moment to install it is before the September policy shifts start moving the crawl numbers anyway.
