AI Crawler Access Checker

Most robots.txt files were written with Google in mind and never revisited. This one reads yours the way an AI crawler would, and tells you what every allow and disallow in it is actually doing.

The names are the problem

There’s no single switch for AI bots. There are about a dozen, they do very different things, and the naming makes them easy to mix up.

Block GPTBot and you stay out of OpenAI’s training data. That says nothing about whether ChatGPT can cite you, which is OAI-SearchBot for indexing and ChatGPT-User for live lookups. Google-Extended governs Gemini training and nothing else, so blocking it won’t touch your Google rankings and allowing it won’t help them. Plenty of sites end up blocking a crawler they meant to allow, and the names are why.

What it checks

  • Your live robots.txt, parsed with longest-match precedence, * wildcards, and $ anchors, the way a compliant crawler would read it
  • Sixteen crawlers in three buckets. AI training: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent. AI search and assistants: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User. Traditional search: Googlebot, Bingbot, DuckDuckBot.
  • Whether you have an llms.txt
  • Whether your robots.txt declares a sitemap

Where this stops

robots.txt is a request, not a firewall. A bot that ignores it crawls you anyway, and a WAF can block a bot you explicitly allowed. If you want to know what your server actually does when these crawlers show up, that’s the Crawler Reality Check.

This fetches your live robots.txt and evaluates it per RFC 9309 (longest-match rules, wildcards, group merging) against each crawler's documented user-agent token. It also looks for an llms.txt file. Nothing is stored.