AI crawlers are no longer just a server-log problem.
Microsoft Clarity is now giving publishers a direct view into which bots are hitting restricted parts of their sites, adding robots.txt violation reporting inside its Bot Analytics dashboard. The update gives marketers, SEOs, and publishers a clearer way to separate useful crawler visibility from bot activity that ignores published crawl rules.
Bot Analytics Is Turning Into A Crawl-Control Dashboard
Microsoft Clarity’s latest release adds robots.txt violation detection to Bot Analytics, expanding the tool beyond traffic measurement and into crawler compliance monitoring.
The new reporting shows when bots request URLs that a site has disallowed through robots.txt. Clarity can now display those violations as a share of total bot requests, show violation patterns over time, and let users filter activity by operator, bot name, and activity type.
That is a meaningful shift for teams already watching AI visibility closely.
For years, most analytics platforms treated bot traffic as noise. Human sessions mattered. Bot visits were filtered, estimated, or pushed into technical log analysis. The rise of AI search, answer engines, and automated content retrieval has changed that. Publishers now need to know not only whether bots are arriving, but which systems are accessing which pages and whether those systems respect the rules set at the site level.
Microsoft’s announcement positions the feature around that exact gap. Clarity checks bot requests against a site’s robots.txt directives and separates compliant from non-compliant activity inside the dashboard.
For teams tracking AI search visibility, that makes Bot Analytics less of a traffic counter and more of a control audit.
Robots.txt Violations Get Their Own Signal
The central addition is a Violations card inside Clarity’s Bot Analytics dashboard.
That card shows violations as a percentage of total bot requests. It is designed to give site owners a quick read on how much automated activity is ignoring crawl preferences, rather than forcing teams to infer non-compliance from raw logs or CDN data.
Clarity also adds a violation trendline, giving teams a way to spot spikes, persistent patterns, or sudden changes in crawler behaviour. That matters because bot activity can vary sharply around content launches, breaking news coverage, product drops, documentation changes, or AI model retrieval cycles.
The dashboard can be filtered by operator, bot name, and activity type. That level of segmentation matters because “AI bot traffic” is not one thing.
Some crawlers support search indexing. Some retrieve content for AI-generated answers. Some are tied to training, verification, or assistant activity. Others may be impersonating known bots or accessing content in ways that do not match a publisher’s policy.
By surfacing violations at the operator and bot level, Clarity gives technical SEO teams a more usable starting point for investigation. The question is no longer just “Are bots crawling us?” It becomes: which bots, which paths, how often, and against which stated rules?
That is a different operating model.
The Most Valuable View May Be The URL List
The most practical part of the release may be content-level visibility.
Microsoft says Clarity lets users review the URLs, paths, and content types generating robots.txt violations. That gives publishers a way to see whether non-compliant bots are attempting to access high-value editorial pages, restricted resources, commercial landing pages, internal search paths, or other areas intentionally placed outside normal crawl access.
For SEOs, that URL-level layer is where the update becomes actionable.
A high violation rate against low-value utility paths may point to a technical cleanup issue. Repeated attempts against monetized content, premium archives, product feeds, or proprietary resources may deserve closer review. A sudden spike against newly published editorial content may indicate that AI crawlers are discovering content faster than existing monitoring tools can explain.
This also brings robots.txt back into the centre of AI content governance.
Robots.txt has always been a directive mechanism, not a security system. It tells compliant crawlers which areas they should not request. It does not physically block access on its own. Sites that need stronger enforcement still rely on authentication, server rules, CDN controls, rate limiting, or other infrastructure-level protections.
Clarity does not change that.
What it does change is observability. Marketers and publishers can now see when the directive layer is being ignored, which can inform whether stricter controls are needed elsewhere.
That distinction is especially important as brands assess robots.txt rules for AI crawlers and compare them with newer proposals like llms.txt, which remains separate from established search crawling systems.
CDN Connections Decide Who Sees The Data
There is a setup requirement.
To use the new violation reporting, a project admin must connect a supported CDN through the AI Visibility section in Microsoft Clarity project settings. Microsoft lists Fastly, Amazon CloudFront, and Cloudflare as supported CDNs.
That requirement makes sense. Robots.txt compliance analysis depends on request-level data. Standard client-side analytics scripts are not enough because they mainly capture browser behaviour after a page loads. Bot requests often never run analytics JavaScript, and many crawlers interact with content at the network or server level.
CDN integration gives Clarity a closer view of automated access patterns.
WordPress sites using the latest Microsoft Clarity plugin get a smoother path. Microsoft says AI Bot Activity is available automatically for sites running the latest plugin. Sites on older versions of the Clarity plugin for WordPress need to update before accessing the feature.
That split is worth watching for small and mid-sized publishers. Many WordPress sites have Clarity installed but may not have Bot Analytics configured at the infrastructure level. Others may rely on Cloudflare, CloudFront, or Fastly already, making the setup less disruptive.
The feature is now available in Clarity for users who already have Bot Analytics configured.
AI Visibility Now Includes Whether Bots Follow The Rules
Microsoft has been steadily pushing Clarity beyond heatmaps and session recordings into AI visibility reporting.
That broader direction matters because search measurement is fragmenting. Traditional ranking reports, click data, and referral traffic are no longer enough to explain how content moves through AI-mediated discovery. A page can be crawled, summarized, cited, ignored, blocked, or accessed in ways that never resemble a conventional organic visit.
TechWyse has already covered Microsoft’s broader push into Bing Webmaster Tools AI reporting, where visibility, citation share, and generative engine optimization signals are becoming a larger part of the reporting conversation.
Clarity’s robots.txt violation reporting adds a different layer: compliance.
For marketers and SEOs, the practical implication is straightforward. Teams can use Bot Analytics to compare compliant and non-compliant crawler activity, identify the pages attracting unauthorized bot requests, and decide whether robots.txt is enough or whether server-side controls need to be reviewed. The data does not prove content misuse on its own. It does give teams a cleaner audit trail for investigating AI bot traffic before making access, crawl budget, or content-protection decisions.
That kind of signal is likely to become more important as publishers separate different forms of AI access. Blocking every bot can reduce discoverability. Allowing every bot can create governance and commercial risks. The middle ground depends on knowing what is happening.
Clarity is trying to put that middle ground inside the analytics dashboard.
The Reporting Gap Is Getting Smaller
Microsoft’s move also reflects a wider industry pattern: AI search tooling is starting to expose the mechanics behind automated access, not just the outcomes.
Google has moved toward more dedicated reporting around generative AI visibility in Search Console. Microsoft has been building AI visibility and citation reporting across Clarity and Bing Webmaster Tools. Publishers, meanwhile, are testing bot policies, crawler blocks, and content-access rules while trying not to cut themselves off from legitimate discovery channels.
The new Clarity feature does not settle the bigger debate over AI crawlers, publisher control, or compensation.
It does give site owners a more specific question to answer.
Not “Are AI bots visiting?”
Which ones are ignoring the rules?
