For years, the robots.txt question founders could answer was "which bots did we allow?" The question they couldn't answer was the one that mattered: "which bots actually listened?" On June 23, Microsoft quietly shipped an answer, for free, inside a tool a lot of SaaS teams already run.
Clarity's Bot Analytics dashboard now surfaces robots.txt violations: which bots ignored your directives and what content they went after. In an era where your AI visibility depends on welcoming the right crawlers while the wrong ones impersonate them, that's the missing audit layer.
Key Takeaways
- Microsoft Clarity's Bot Analytics now reports robots.txt violations: violation share of total bot requests, trends over time, targeted paths, and compliant vs non-compliant volumes per bot.
- It works from server-side CDN logs, sidestepping the obvious problem that bots don't run your JavaScript tag; you'll need to connect Cloudflare, Fastly, CloudFront, Azure Front Door, or Akamai first.
- The context is ugly: documented stealth crawling by Perplexity, Bytespider ignoring disallow rules at ~10% of AI bot traffic, and Cloudflare finding roughly a third of "GPTBot" traffic is spoofed impostors.
- The right response is rarely "block everything": verify identity first, keep the compliant retrieval crawlers that feed your AI visibility, and WAF-block the fakes.
- Cloudflare's September 15 default-blocking deadline makes this quarter the time to make deliberate crawler decisions instead of inheriting someone else's defaults.
What Shipped
The announcement came from the Clarity team on June 23, with the kind of plain-spoken framing you rarely get from analytics vendors:
𝗡𝗘𝗪: Clarity's Bot Analytics dashboard now surfaces robots.txt violations. 🤖 You can now see exactly which bots are ignoring your crawl rules and what content they're trying to access. → Violations as a % of total bot requests → Violation trends over time → Filter by https://t.co/LBDGbGIiRP
Microsoft Clarity@msftClarityJun 23, 2026
Their summary, preserved in paraphrase: you can now see exactly which bots are ignoring your crawl rules and what content they're trying to access, so you know which bots are playing by the rules and which ones aren't. The Violations card lives under Bot Analytics in the AI Visibility section and reports four things: violations as a percentage of total bot requests, a violation trendline, the specific paths being hit against your directives, and a side-by-side of compliant versus non-compliant request volume. Filters cover operator, bot name, and activity type, with the named roster spanning GPTBot and ChatGPT-User, the ClaudeBot family, Gemini, Copilot, and PerplexityBot.
The clever part is how it sees any of this. Bots don't execute your analytics JavaScript, so tag-based tools are structurally blind to them. Clarity instead ingests server-side logs from a connected CDN and checks each request against your robots.txt at read time. Which also explains the setup catch: nothing appears until a project admin connects a supported CDN (Cloudflare, Fastly, Amazon CloudFront, Azure Front Door, or Akamai) in project settings. WordPress sites get it wired automatically through the current Clarity plugin.
One thing I should've mentioned earlier: this is the fourth AI-visibility feature Clarity has shipped since mid-May, after the AI citations report, the expanded bot dashboard, and topic insights. Microsoft is methodically turning a free heatmap tool into the default instrumentation layer for AI-era traffic, and at the price of zero, it's working.
Why Compliance Visibility Suddenly Matters
A robots.txt file has always been a request, never a lock. What changed is the volume and the audacity of the parties ignoring it.
The rap sheet, documented rather than rumored: Cloudflare caught Perplexity stealth-crawling sites that had blocked it, using disguised browser user agents and IPs outside its published ranges. ByteDance's Bytespider routinely ignores disallow directives and has grown to roughly a tenth of all AI bot traffic. And Cloudflare's 2026 verification data found that about a third of traffic identifying as GPTBot was spoofed scraping infrastructure wearing OpenAI's name. Meanwhile AI crawling has become the majority of crawler traffic: 52% of crawler requests are now AI-related, up from 22% a year earlier.
Fair warning though: the violation percentages you'll see in Clarity need interpretation before outrage. A "GPTBot violation" is frequently not OpenAI misbehaving; it's an impostor borrowing the user agent, which is precisely why the verification step below matters more than the dashboard's red numbers.
The broader ecosystem is moving from measurement to enforcement on a schedule. Cloudflare declared July 1 its "Content Independence Day", splitting crawlers into search, agent, and training categories, and from September 15 new domains get training and agent crawlers blocked by default. Deliberate or not, your crawler policy is about to be set. Better it be set by you.
AI search traffic grew 835% this year. Is your content ready?
RankControl generates 15+ content types optimized for ChatGPT, Claude, and Perplexity. Published on your domain, matched to your brand.
The AEO Tension: You Want (Some of) These Bots
Here's the thing that separates this blog's take from the "block all AI bots" reflex: for a SaaS company competing on AI visibility, compliant crawlers are the distribution channel. OAI-SearchBot fetching your comparison page is how you end up cited in a ChatGPT answer. Blocking retrieval crawlers to punish training scrapers is burning the storefront to stop shoplifting, a distinction we mapped bot-by-bot in our llms.txt and robots.txt checklist.
What the violations report finally enables is policy by evidence. You can hold a generous allow-list for the crawlers that earn you citations, and know within a week if something claiming to be one of them is treating your directives as decoration. The SEO community's early read matches ours; the feature made the weekly news roundups with practitioners noting it'll become a standard part of technical audits:
SEO News: Google June 2026 spam update rolls out in two days, Microsoft Clarity adds robots.txt violation tracking to Bot Analytics, Google Search Console AI performance reports expand beyond the UK
The SEO world is always full of surprises, so let's stay on top of the latest events with our news digests: Updates Google June 2026 spam update rolls out in two days The June 2026 spam update began on June 24 and completed June 26. Global,...
And the sharpest distribution of the news came through Aleyda Solis's SEOFOMO digest, which is how most of the SEO world found out:
🚨 A new Spam Update and More SEO & AI Search News [From #SEOFOMO, June 28, 2026] 👇 * Google released and completed the rollout of the June 2026 Spam Update * Microsoft Clarity Now Surfaces Robots.txt Violations in Bot Analytics * Google’s Mueller Explains How AI Search https://t.co/UGR2BX3LAE
Aleyda Solis 🕊️@aleydaJun 28, 2026
Built by the team that got cited in 48 hours.
Content generation, backlink building, AI visibility tracking, and lead capture. One platform, zero guesswork.
The Response Checklist
Five steps, about an hour for the first pass:
- Connect your CDN in Clarity. Project settings, AI Visibility, pick your provider. No CDN connection, no data. (If you're on Cloudflare, this pairs with their own crawl-control panel.)
- Read a week of violations by operator. Sort by violation share, note which bots and which paths. Expect surprises; most teams have never seen this layer of their own traffic.
- Verify identity before judging behavior. The major vendors publish their crawler IP ranges (OpenAI's gptbot.json, Anthropic's and Perplexity's equivalents), and forward-confirmed reverse DNS separates the real crawler from the costume. A "ClaudeBot" arriving from a residential proxy is not Anthropic.
- Act by category. Verified retrieval crawlers behaving well: leave them alone, they're your AI visibility. Unverified impostors: block at the WAF; Cloudflare's managed AI-scraper rules and verified-bot checks handle most of it. Genuine training crawlers: a strategy decision, not a security one, and we've made the case that training-data inclusion is a long-term brand asset for most SaaS companies.
- Recheck monthly, and after every bot-policy change. One footnote from Google's John Mueller this month applies here: Cloudflare's new content-signals robots.txt directive currently has "no effects whatsoever" on any crawler, so treat directives as requests you audit, never controls you trust.
The pattern underneath this feature is the same one we keep landing on: access is the first gate of AI visibility, and it's now measurable end to end. Clarity tells you who's reading your site and whether they listened, while AI visibility tracking tells you whether the reading turned into citations. Pair both with a content engine that keeps the compliant crawlers fed with something worth quoting, and the whole chain from access to answer is under observation. The teams that lose this era won't be the ones who blocked the wrong bot once. They'll be the ones who never looked.
See your first AI citation report in under 5 minutes.
No setup calls. No onboarding meetings. Connect your domain and see where AI mentions your brand right now.
