Crawler Reality Check

Your robots.txt says what you'd like crawlers to do. Your server decides what actually happens to them. This fetches your URL with real crawler user-agents and compares every response against a normal browser.

The failure this catches

It’s the most common serious technical SEO incident of the CDN era. A WAF rule, a bot-fight setting, or a rate limiter starts handing Googlebot 403s and challenges. Traffic slides for weeks before anyone connects it to the change, because the page loads perfectly well in your browser, which is exactly why nobody catches it. The same misconfiguration now quietly decides whether AI assistants can read and cite you at all.

What it compares

It runs a normal browser request plus five crawlers (Googlebot Smartphone, Bingbot, GPTBot, ClaudeBot, PerplexityBot) and lines up:

  • Final status code and the whole redirect chain
  • Title, meta robots, and X-Robots-Tag. A noindex served only to bots is a classic accidental cloak.
  • Canonical URL
  • Response size, as a rough signal that bots are being handed different content

Reading the results

A 403 or a challenge served to a spoofed Googlebot user-agent doesn’t prove real Googlebot is blocked. Good bot protection verifies crawlers by IP, and these requests come from Cloudflare’s edge rather than Google’s network. The tool labels those cases separately and points you at your server logs or Search Console’s Crawl Stats, which is where you’d want to confirm it anyway.

Requests go out from Cloudflare's edge with each crawler's published user-agent string. Sites that verify crawlers by IP address, which is what Google and others recommend, may challenge these requests even though they'd serve the real bot fine. The tool flags that case instead of calling it a block. Nothing is stored.