Do you know which bots ignored your robots.txt last week? Until June 23 you almost certainly didn't, because the only question the file ever let you answer was which bots you had allowed. That day Microsoft quietly shipped the missing half, for free, inside Clarity, a tool plenty of SaaS teams already have installed.
The feature is easy to describe. Clarity's Bot Analytics dashboard now reports robots.txt violations, meaning which bots ignored your directives and which content they went after. It matters more than it sounds. Your AI visibility depends on letting the right crawlers in while the wrong ones pretend to be them, and until now there was no audit layer for that.
What shipped
Microsoft has been busy. Since mid-May, Clarity has added the AI citations report, the expanded bot dashboard and topic insights, and this is the fourth AI-visibility feature in that run. I read it as a deliberate plan to make a free heatmap tool the default way to instrument AI-era traffic, and since it costs nothing, the plan is working.
The Clarity team's June 23 announcement was plainer than most vendor launches:
𝗡𝗘𝗪: Clarity's Bot Analytics dashboard now surfaces robots.txt violations. 🤖 You can now see exactly which bots are ignoring your crawl rules and what content they're trying to access. → Violations as a % of total bot requests → Violation trends over time → Filter by https://t.co/LBDGbGIiRP
Microsoft Clarity@msftClarityJun 23, 2026
Their message, paraphrased, is that you can now see exactly which bots ignore your crawl rules and what content they're after, so you know who plays by the rules. It lives in a Violations card under Bot Analytics, inside the AI Visibility section:
| Piece | What it shows |
|---|---|
| Violation share | Violations as a percentage of total bot requests |
| Trend | A violation trendline over time |
| Paths | The specific paths bots hit against your directives |
| Compliance split | Compliant versus non-compliant request volume, side by side |
| Filters | Operator, bot name and activity type |
| Named bots | GPTBot and ChatGPT-User, the ClaudeBot family, Gemini, Copilot, PerplexityBot |
| Data source | Server-side logs from a connected CDN: Cloudflare, Fastly, Amazon CloudFront, Azure Front Door or Akamai |
The row I'd point your engineers to is the data source. Bots never execute your analytics JavaScript, so a tag-based tool can't see them at all. Clarity reads the server-side logs from your CDN instead and checks each request against your robots.txt at read time. That's why the report stays blank until a project admin connects a supported CDN in project settings. If you're on WordPress, the current Clarity plugin wires this up automatically.
Why compliance visibility suddenly matters
The date I'd circle is September 15. Cloudflare declared July 1 its "Content Independence Day" and split crawlers into search, agent and training categories, and from September 15 new domains get training and agent crawlers blocked by default. Enforcement is arriving on a timetable, so your crawler policy is about to be set whether you decide it or not. I'd rather you set it than inherit it.
To set it well you need evidence, because robots.txt has only ever been a request and more crawlers now ignore it, some of them brazenly. Cloudflare caught Perplexity stealth-crawling sites that had blocked it, using disguised browser user agents and IPs outside Perplexity's published ranges. ByteDance's crawler, Bytespider, routinely ignores disallow directives, and it has grown to roughly a tenth of all AI bot traffic.
Scale makes it worse. AI crawling has become the majority of crawler traffic, with 52% of crawler requests now AI-related against 22% a year earlier, so AI crawlers are most of the bot traffic this report watches.
The finding that bothers me most is about GPTBot. Cloudflare's 2026 verification data showed that about a third of the traffic calling itself GPTBot was spoofed scraping infrastructure using OpenAI's name. So read Clarity's percentages before you get angry at them. A "GPTBot violation" is often an impostor borrowing the user agent, with OpenAI nowhere near it, which is why the verification step below matters more to me than any red number on the dashboard.
AI search traffic grew 835% this year. Is your content ready?
RankControl generates 26 content formats optimized for ChatGPT, Claude, and Perplexity. Published on your domain, matched to your brand.
The AEO tension: you want (some of) these bots
After reading all that, your first instinct might be to block every AI bot you can name. I'd talk you out of it. If you're a SaaS company competing on AI visibility, compliant crawlers are your distribution channel, and OAI-SearchBot fetching your comparison page is how you end up cited in a ChatGPT answer. Block the retrieval crawlers to punish the training scrapers and you cut off the very visits that were getting you cited. Our llms.txt and robots.txt checklist goes through the difference bot by bot.
The violations report is what lets you hold that line with evidence. Keep a generous allow-list for the crawlers that earn you citations, and you'll still find out within a week if something claiming to be one of them treats your directives as decoration. The SEO community got to the same read early, and the feature made the weekly news roundups with practitioners saying it'll become a standard part of technical audits:
SEO News: Google June 2026 spam update rolls out in two days, Microsoft Clarity adds robots.txt violation tracking to Bot Analytics, Google Search Console AI performance reports expand beyond the UK
The SEO world is always full of surprises, so let's stay on top of the latest events with our news digests: Updates Google June 2026 spam update rolls out in two days The June 2026 spam update began on June 24 and completed June 26. Global,...
Most people in SEO actually heard about it through Aleyda Solis's SEOFOMO digest, which gave the news its widest reach:
🚨 A new Spam Update and More SEO & AI Search News [From #SEOFOMO, June 28, 2026] 👇 * Google released and completed the rollout of the June 2026 Spam Update * Microsoft Clarity Now Surfaces Robots.txt Violations in Bot Analytics * Google’s Mueller Explains How AI Search https://t.co/UGR2BX3LAE
Aleyda Solis 🕊️@aleydaJun 28, 2026
Built by the team that got cited in 48 hours.
Content generation, backlink building, AI visibility tracking, and Google rankings. One platform, zero guesswork.
The response checklist
One wrong block is easy to undo. Going all year without opening this report is the mistake I'd most want you to avoid, so give the first pass about an hour and work through five steps:
- In Clarity's project settings, open AI Visibility and connect your CDN provider.
- Wait a week, then read the violations by operator, sorted by violation share.
- Verify each suspect's identity before you judge its behavior.
- Act by category.
- Recheck monthly and after every bot-policy change.
Step one is non-negotiable, since no CDN connection means no data (on Cloudflare, it pairs with Cloudflare's own crawl-control panel). In step two, note which bots hit which paths, and expect surprises, because most teams have never seen this layer of their own traffic.
Step three is where the value sits. The major vendors publish their crawler IP ranges (OpenAI has gptbot.json, and Anthropic and Perplexity publish equivalents), and forward-confirmed reverse DNS tells a real crawler apart from an impostor wearing its name. A "ClaudeBot" arriving from a residential proxy is not Anthropic.
For step four, leave verified retrieval crawlers that behave well alone, because they are your AI visibility. Block the unverified impostors at the WAF, where Cloudflare's managed AI-scraper rules plus its verified-bot checks handle most of the work. Genuine training crawlers are a strategy call more than a security one, and we've argued that training-data inclusion is a long-term brand asset for most SaaS companies.
Step five matters because the directives themselves are softer than they look. Google's John Mueller added a footnote this month: Cloudflare's new content-signals robots.txt directive currently has "no effects whatsoever" on any crawler. Treat directives as requests you audit, never as controls you trust.
Access is the first gate of AI visibility, and you can finally measure it end to end. Clarity shows you who read your site and whether they listened. AI visibility tracking shows whether that reading turned into citations, and a content engine keeps the compliant crawlers supplied with something worth quoting, so you can watch the whole chain from access to answer.
See your first AI citation report in under 5 minutes.
No setup calls. No onboarding meetings. Connect your domain and see where AI mentions your brand right now.
