Cloudflare hid my site from AI and I sell AI visibility
I help local businesses show up when somebody asks an AI assistant for a recommendation. In August I pointed my own tooling at my own website and it came back 403. Not to browsers, only to GPTBot, ClaudeBot and PerplexityBot. My robots.txt also carried Disallow: / for eight AI crawlers and a Content-Signal: ai-train=no. I had written none of it.
It came from a Cloudflare setting live on my account, and my site had been unreadable to every assistant for its entire existence, along with all 36 demo pages I had been sending to prospects.
Two things below were already documented before I started, and I discovered neither of them. The split between training crawlers and search-time agents is documented by the vendors themselves. GPTBot, ClaudeBot and PerplexityBot collect pages in bulk to train on. OAI-SearchBot, ChatGPT-User, Perplexity-User and Claude-SearchBot fetch a page at the moment a person asks a question. Blocking the first group does not make you unrecommendable.
Large-scale measurement of AI crawler blocking also exists. "Consent in Crisis: The Rapid Decline of the AI Data Commons" (Data Provenance Initiative, NeurIPS 2024) audits robots.txt across the web domains behind C4, RefinedWeb and Dolma, which is far more of the web than I touched.
Neither has been applied to local business websites, which are the actual subject of "will an assistant recommend me". So that is what I measured, and the useful result was not the one I went looking for.
Why my first scan said zero
I scanned 29 local business websites, found nothing blocking, wrote the idea off and moved on. Weeks later I found the bug that produced that zero: my robots.txt parser could not see User-agent: *, so any site whose file said
User-agent: *
Disallow: /came back as open, and that is the single most common way a site blocks everything. Then a second bug. The scanner read robots.txt and status codes only, so it could not see a noindex tag, and since WordPress 5.3 the "Discourage search engines" setting stopped writing Disallow: / and started emitting noindex instead. A robots.txt-only scanner is blind to every WordPress site that ticked that box.
Then the worse problem: I had never saved the list of 29 hosts. No CSV, no text file. That scan can never be re-run or checked by anyone, including me, and I cannot correct a number I cannot check, so I retired it. I re-ran the fixed scanner against 310 local business sites whose host list I wrote to disk before running anything, and 293 answered.
What the 293 sites showed
Fifteen refuse an AI crawler in robots.txt, and six of those serve the same file, byte for byte, across six different dental practices:
User-agent: Googlebot
User-agent: bingbot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-SearchBot
Disallow: /~*
User-agent: *
Disallow: /Every search-time agent is allowed. The training crawlers are not named, so they fall through to the blanket disallow. Somebody stayed out of the training sets while staying readable when a customer asks, and fourteen of the fifteen robots.txt blockers draw exactly that line. Fifteen out of 293 is 5.1%. Reported as one number, "5% of sites block AI" describes almost nothing that costs a business a customer, which is why I stopped treating a robots.txt count as the finding.
The finding was somewhere else. Separately from robots.txt, twenty sites refused an AI crawler while serving browsers, Googlebot, an unknown agent and curl normally. Eighteen of those refusals held when I re-tested one request at a time, three seconds apart.
Eleven refuse with a 403. The other seven return a 429 that never lets up, a rate-limit code used as a door. I count those separately because they are weaker evidence and my first pass discarded 429s on principle, but these seven held on every re-test while a browser got 200 from the same address moments later.
Twelve of the eighteen have no robots.txt rule about AI at all. On the stricter bar, counting only the 403s, it is five of eleven. Either way, most of the sites doing the blocking never wrote it down, and that gap is the whole thing.
The two groups also split cleanly. Every site that wrote an AI rule returns a plain 403. Every one of the seven persistent 429s has no rule at all. Deliberate refusal announces itself and answers 403; the silent kind shows up as a rate limit.
When a business means to opt out it says so in the file, the way those six dental practices did. These twelve say nothing anywhere, and something at the edge still turns crawlers away. They span six different setups. I re-read their Server headers on 1 September 2026: four sit behind Cloudflare, three on nginx, two on LiteSpeed, one Apache, one a CDN calling itself only hcdn, and one that sends no Server header at all. Three of the nginx sites identify as WP Engine, which is the nearest thing to a single culprit and is still only a quarter of them, so no one vendor's default explains this. What explains it is bot-protection products treating AI crawlers as something to stop, on sites whose owners have never opened the setting.
Stated policy and actual behaviour are different measurements, and only one of them is easy to collect. Every robots.txt survey I have read measures the easy one.
What I ruled out, including my own mistakes
A flaky 429 is not a policy. My first pass found 42 apparent blocks, which felt too high, so I re-tested each with one request three seconds apart and no concurrency. Four were 429-only and moved between runs, three stopped refusing altogether, and since the original scan made eight requests per host across ten workers, a share of those rate limits was me. The seven 429s I do count held every single time.
I also had to rule out the possibility that these CDNs refuse anything unfamiliar, because a CDN that allowlists known-good bots looks identical to a deliberate AI policy. My scanner defined an unknown-agent control and never used it, so the comparison its own docstring promised had never run. I ran it, probing every blocker with an unremarkable unknown agent and with curl, and not one refused them. The refusals are agent-specific.
The last mistake was mine and it was in the product. The plugin I ship labelled GPTBot as "ChatGPT". So a business that deliberately opted out of training got told "ChatGPT: blocked".
That reads as "you are invisible", when the truth is closer to "you chose not to be training data and you are still perfectly recommendable." I have shipped a fix. The point stands: I missed the distinction in a tool whose whole job is not missing it.
Method, and how to check your own site
The 310 hosts came from OpenStreetMap across Fairfax, Loudoun and Arlington in Virginia, covering seven trades (dentists, hairdressers, car repair, veterinary, HVAC, plumbers and roofers) plus two petrol stations that came in on the car repair query and that I left in instead of quietly dropping.
Each site was fetched using each crawler's real user agent string, not a shortened token, because firewall rules match the whole string, and the controls were an ordinary browser, Googlebot, an unknown agent and curl. robots.txt was read from the domain root and parsed with group precedence, so a * rule applies only when no group names the crawler, and an Allow cancelling a Disallow is respected. The host list was written to disk before anything ran, a rule that exists because of the 29 I lost.
To check your own site, fetch your front page as an AI crawler and see what comes back:
curl -s -o /dev/null -w "%{http_code}\n" -A \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" \
https://yoursite.com/A 200 means you are readable. A 403 or 429 means something is turning them away, and if your robots.txt does not mention AI crawlers, you did not ask for it.
Then read the file itself. If the only names in your disallow are GPTBot, ClaudeBot and PerplexityBot, you have opted out of training and assistants can still read you when someone asks, which may be exactly what you wanted. If OAI-SearchBot or ChatGPT-User are in there, that is the one worth a second look.