Microsoft Clarity Adds Bot Violation View for Robots.txt Compliance

Microsoft Clarity has added a new Bot Analytics feature that surfaces bot requests that violate a site’s robots.txt directives. The feature calculates the percentage of total bot activity that hits disallowed paths and displays it in the dashboard, allowing site owners to filter by bot operator, name, activity type, URLs, and paths. Clarity also offers a side‑by‑side view that compares compliant crawlers with violators, helping users identify patterns in crawler behavior. The feature must be enabled manually in the AI Visibility settings and is available only for sites using supported CDNs such as Fastly, CloudFront, Cloudflare, Azure Front Door, and Akamai, as well as WordPress sites with the latest Clarity plugin. Clarity’s new view does not explain why a request violated the rule, but it provides a clear record of disallowed activity that can be used to adjust robots.txt or block offending bots. The addition addresses growing concerns about AI crawlers consuming server resources and skewing analytics, offering a no‑cost way to monitor compliance. By automatically counting disallowed requests, Clarity reduces the need for manual log parsing and testing of URLs against robots.txt.