Test your robots.txt against GPTBot, ClaudeBot, PerplexityBot, Googlebot, and every other crawler that decides whether AI can cite you.
Free, no login, no email. Nothing stored beyond a 24 hour cache. API and MCP
Paste your URL
We fetch the robots.txt at your domain root and parse it exactly the way a crawler does.
We test every AI crawler
Eleven answer engines and six training crawlers, each evaluated against the path you gave us, with the deciding rule shown.
We test your firewall too
We request your page as GPTBot and compare it to a browser request, because a CDN can block crawlers your robots.txt welcomes.
The first is the one everybody knows: a rule in robots.txt. The second is the one that catches people out. A CDN can block crawlers at the edge, before the request reaches your server. Your robots.txt says come in. Cloudflare says no. Nothing in your CMS or your SEO plugin reports this, and the only way to find it is to send a request as a crawler and see what comes back. That is the firewall test on this page.
This became common quickly. Cloudflare added a one-click block for AI scrapers and a lot of site owners turned it on, reasonably, without realising the same switch removes them from ChatGPT and Perplexity answers. If you have ever wondered why an AI assistant cannot see a site that is plainly public, this is usually why.
Three rules decide everything, and most checkers implement at least one of them wrong.
One group applies, not all of them. A crawler picks the group whose User-agent token is the longest match for its own name, and follows only that group. Everything under User-agent: * is ignored the moment a more specific group for that crawler exists anywhere in the file. This is why adding an empty User-agent: GPTBot group accidentally unblocks GPTBot from every wildcard rule above it.
The longest match wins, not the first. Rule order is irrelevant. Disallow: /blog/ followed by Allow: /blog/public/ allows the public folder, because that pattern is longer and therefore more specific.
Ties go to Allow. When an Allow and a Disallow match a path with the same specificity, the Allow wins. An empty Disallow: means nothing is disallowed, not that everything is.
We split them into two groups because the decision is different for each. Answer crawlers, like OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot, decide whether an AI system can find you and cite you. Training crawlers, like CCBot, Google-Extended, Applebot-Extended, and Meta-ExternalAgent, only collect data for model training.
Blocking the second group is a legitimate editorial decision that costs you nothing in AI answers, and this tool reports it as a note rather than an error. Blocking the first group is the thing worth knowing about.
Paste your URL above. We fetch your robots.txt, parse it the way a real crawler does, and evaluate every AI user agent against the exact path you gave us. The results table shows each crawler, whether it is allowed or blocked, and which rule decided it.
The same run this page does, as an HTTP endpoint and as the MCP tool check_robots_txt_for_ai. No account, no API key. Full reference.
curl -sS https://usesuperflow.ai/api/tools/robots-txt-ai-checker \
-H 'Content-Type: application/json' \
-d '{"url":"example.com"}'Returns { ok, report: { accessScore, crawlers[] with the rule that decided each verdict, firewall, findings[] }, cached, ageSeconds }. 10 runs per hour per IP. Allow up to 75 seconds.
claude mcp add --transport http superflow https://usesuperflow.ai/api/mcpThen ask your agent for check_robots_txt_for_ai. Setup for Claude Desktop, Cursor and VS Code is on the reference page.
No login, no email, no ads. Same as this one.
Superflow agents watch every page of every site you ship and catch the change that broke something, before your client does.