I Scanned 1,207 Agency Robots Files. Two Were Actually Blocking AI Answers.

I scanned the robots.txt file of 1,207 marketing and SEO agency domains to find sites that had accidentally hidden themselves from AI answer engines. I expected a healthy list. The pitch writes itself: your competitors are invisible to ChatGPT and they do not know it.

Two sites out of 1,207 were actually hiding from an answer engine.

The AI-blocking problem that a growing number of tools now audit, score, and invoice for is, at least across this population, close to nonexistent. Here is the full count, the method, and the mistake I made first.

The numbers#

Of 1,207 domains, 1,091 returned a readable robots.txt. The rest failed in ways that do not indicate blocking: 57 served an HTML error page where the text file should be, 40 returned an HTTP error, 16 failed to resolve, 3 timed out. A missing robots.txt permits everything, so the true block rate is if anything lower than what follows.

Of the 82 sites blocking a training crawler, 37 are running Cloudflare's managed default. That is 45 percent of all visible AI-blocking on these sites, produced by a setting the site owner did not write and in most cases did not choose.

The most-refused agents are training crawlers by a wide margin: Bytespider 64, CCBot 64, GPTBot 54, Google-Extended 48, meta-externalagent 48, Amazonbot 47, ClaudeBot 47, Applebot-Extended 45.

The distinction the whole thing turns on#

There are two disjoint sets of AI crawlers, and conflating them is the error that makes this problem look large.

Training crawlers collect text to train a future model. GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Bytespider, Amazonbot, meta-externalagent, Applebot-Extended. Refusing them is a licensing position. It has no effect on whether an assistant can quote you in an answer today.

Retrieval crawlers fetch your page because a person just asked a question. ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-Web, Claude-SearchBot. A Disallow under one of these names is the only robots.txt rule that removes a site from live AI answers.

An audit that counts a GPTBot block as an AI-visibility defect is reporting a licensing decision as a bug. The site owner usually made that decision on purpose.

The Cloudflare default is well designed, which is inconvenient for the pitch#

Cloudflare injects a block into robots.txt marked with the comment BEGIN Cloudflare Managed content. It looks like this:

# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
 
User-agent: Amazonbot
Disallow: /
 
User-agent: Applebot-Extended
Disallow: /
 
User-agent: Bytespider
Disallow: /
...
# END Cloudflare Managed Content

Read what it does. Everything is allowed. The Content-Signal line states the position in words: yes to search, no to training, reference use permitted. Then a fixed list of training crawlers is refused by name.

Across all 37 Cloudflare-managed sites in the census, zero blocked a retrieval crawler. Not one. The default is precisely targeted at the thing a publisher would actually want to refuse, and it leaves every live-answer path open.

Cloudflare regenerates that section, which means two things. The owner did not choose it, and the owner cannot fix it by editing the file. Both facts make it look like a finding to a scanner and neither makes it a problem.

There is a tell for this. Three unrelated agencies in my first pass produced hits at identical line numbers, 45 and 54. Identical output from unrelated inputs is never a finding about the inputs. It is a fact about a shared generator, and it should be read as a reason to distrust the instrument.

The twelve sites that do block retrieval#

Naming them, because a census that reports a rate without the underlying rows is asking to be trusted rather than checked.

Ten of the twelve are publishers and platforms taking a deliberate editorial position. USA Today, Yahoo Finance, SlideShare, the Built In network and a county newspaper are negotiating, and a newspaper refusing an answer engine usually has a lawyer behind the decision. OpenAI's own consumer domain refusing Perplexity and Claude is a competitive stance, not an oversight.

That leaves two agency sites where the block looks unintentional. Two out of 1,207.

I got this wrong first, in the direction that paid better#

My scanner had a list of crawler names in a constant called LIVE. It contained GPTBot, ClaudeBot and anthropic-ai alongside the real retrieval agents. Naming the list LIVE made its contents feel verified. They were not.

That list produced 39 drafted messages telling technical SEO agencies they were hidden from ChatGPT. Thirty-six of those claims were false. Every message shipped a curl command inviting the recipient to check, to an audience that would have run it.

Nothing was sent. The error surfaced because the step that finds a recipient's address ran before the step that composes a message, and the address-finding pass re-fetched each robots.txt. Five of seven came back with identical crawler pairs. That is when the line numbers gave it away.

The interesting part is where the defect lived. Both of my existing parser tests passed. The parser was correct. The wrong thing was a factual claim wearing the costume of a constant, in a file full of code that nobody was going to review as prose. The fix was to split the list in two, name the halves for what they do, and add a test asserting that a training-only block produces no finding.

Check your own site in one command#

curl -s https://yourdomain.com/robots.txt | grep -iE -A2 \
  'ChatGPT-User|OAI-SearchBot|PerplexityBot|Claude-Web|Claude-SearchBot'

If that returns nothing, no answer engine is being turned away and your robots.txt is not your problem. If it returns a Disallow: / under one of those five names, deleting those two lines is the entire fix.

That is worth stating plainly, because the surrounding industry is arranged to suggest otherwise. Refusing GPTBot is a choice about training data and reasonable people make it deliberately. It is not a leak, a misconfiguration, or a reason to buy anything.

What this changes about AI visibility work#

If 92.5 percent of sites block nothing and 98.9 percent leave every retrieval path open, then robots.txt is a solved surface and an audit built around it is auditing a defect almost nobody has. Sites that go uncited by AI answer engines are, in the overwhelming majority, perfectly reachable and not being chosen.

That is a harder problem and a more honest one. It is also the only one left once the file check comes back clean.

Method: robots.txt fetched over HTTPS for 1,207 agency domains on 18 August 2026, with a browser user agent and a 10 second timeout. A domain counts as blocking an agent when a group naming that agent carries Disallow: / without a cancelling Allow: /. Curated allowlists, meaning a root disallow paired with a narrower allow, are not counted as blocks. Robots files that returned HTML are excluded rather than treated as permissive.