Your robots.txt allows everything. Your pages look fine in Chrome. And ChatGPT still describes your product wrong, or skips you entirely while quoting a competitor's spec sheet.
That's usually not a content problem. Whatever produced the answer never saw the page you think you published.
AI crawlers are the least glamorous part of the whole AI-visibility conversation and the part that breaks most often. They're also not one thing. There's a bot that trains models, a bot that fetches pages to cite in answers, and a browser agent working for a human who is on your site right now. Different visitors, different costs when you turn them away, and one robots.txt that usually treats them identically.
Who's actually in your logs: the AI crawlers by name
Every major vendor runs a version of the same three-part pattern: a training crawler you can refuse, a search bot you have to allow if you want to appear at all, and a user-initiated fetcher that may ignore your robots.txt.
| User agent | Whose | Class | What happens if you block it |
|---|---|---|---|
| GPTBot | OpenAI | Training | You opt out of model training. Answer visibility is a separate switch. |
| OAI-SearchBot | OpenAI | Search / retrieval | You disappear from ChatGPT search answers. OpenAI documents this outright. |
| ChatGPT-User | OpenAI | User-initiated fetch | Documented as possibly ignoring robots.txt anyway. |
| ClaudeBot | Anthropic | Training | Training opt-out. |
| Claude-SearchBot | Anthropic | Search / retrieval | You lose the retrieval path into Claude's answers. |
| PerplexityBot | Perplexity | Search / citation | You lose Perplexity citations. |
| Perplexity-User | Perplexity | User-initiated fetch | Also documented as possibly ignoring robots.txt. |
Two honest gaps in that table. Vercel's Agent Readability Spec also names `CCBot` and `Google-Extended` in its don't-block list, and I've left the behaviour column empty for them because I haven't verified those first-hand. Meta fetches pages too, and I'm not going to guess at its user-agent string in a table people will copy into production.
The pair that costs real money is GPTBot and OAI-SearchBot. GPTBot is training. OAI-SearchBot is retrieval, and OpenAI's wording on the consequence isn't subtle: sites that are opted out will not be shown in ChatGPT search answers. A lot of teams blocked "the OpenAI bot" back in 2024 to protect their content, and in doing so removed themselves from the place their buyers now go to ask questions. Nobody sent them a notification about it.
Three kinds of AI crawlers, and why the difference matters
Training crawl
Bulk, indiscriminate, and no referral coming back. Cloudflare found 52% of crawler requests in June 2026 were for training, with mixed-use purposes above 36%.
Retrieval and citation
Smaller share, much sharper spikes. Fastly's May 2026 data splits AI bots roughly 85% crawlers to 15% fetchers, and among the fetchers ChatGPT is 98%, with bursts above 39,000 requests a minute. That's a capacity conversation, not a crawl-budget one, and it tends to land on whoever is on call rather than on whoever owns SEO.
On-demand agent fetches
Someone asks their browser agent to do something and it hits your site live, in the moment, on behalf of a person with a credit card. Cloudflare measured "user action" crawling growing more than 15x during 2025. HUMAN Security's July 2026 snapshot of agentic browsing breaks down as Comet 47.1%, Claude's Chrome extension 24%, ChatGPT Atlas 15.5% (Atlas is being sunset), and ChatGPT Agent 6.1%. 76% of that activity lands on product and search routes, while checkout is 2.6%. Agents research; humans still close.
One scale check before anyone panics. Bots were 62% of HTML page requests in Cloudflare Radar's live data for late August 2026, but AI bots specifically averaged 4.2% of HTML traffic across 2025. Anyone telling you AI agents have overtaken your human audience is selling you something.
The one fact that breaks the most sites: they don't run JavaScript
Vercel's crawler research, and I'll date it because the date matters, November 2024, found that no major AI crawler executes JavaScript. OpenAI, Anthropic, Meta and Perplexity all fetch your JS bundles and never run them.
Sit with that for a second if you run a single-page app. The crawler requests your route, receives an HTML shell with an empty root div, and moves on. Your pricing table, your product copy, your FAQ, all of it arriving after hydration, does not exist as far as that crawler is concerned. You aren't blocked. You're blank.
Two more numbers from the same research and the same date, because they compound. AI crawlers waste about a third of their requests on 404s: ChatGPT at 34.8%, Claude at 34.2%, against Googlebot's 8.2%. And Fastly measured over 51% of AI requests reaching origin, compared with under 9% of human requests, so your CDN cache isn't absorbing these visitors anywhere near the way it absorbs people. Thin content plus a bad link graph plus no cache shielding is a rough combination when the crawler is already giving up on a third of what it asks for.
The test takes thirty seconds: `curl` the page and read what comes back, or run it through `https://r.jina.ai/<your-url>` and look at what an agent sees. If the interesting words aren't in that response, no amount of llms.txt or schema is going to rescue you. We went through the llms.txt evidence separately and the short version is that it mostly doesn't move the needle.
Server-side render the content that matters. That's the fix, it's boring, and it's most of the work.
Blocking AI crawlers: what it actually costs
The case for blocking is real. Cloudflare's July 2025 crawl-to-referral table is the most-quoted evidence and it deserves to be: Anthropic at 38,066 crawls per referral, OpenAI at 1,091:1, Perplexity at 195:1, Google at 5.4:1. The fair caveat is that the Claude app sends no Referer header, and these ratios swing hard between measurements. Even so, if your product is the text on the page, that asymmetry is an argument, and I'd have a hard time telling a publisher to shrug it off.
The case against blocking is a revenue argument rather than a principled one. Shopify's Q1 2026 platform data: AI-referred shoppers convert about 50% higher with 14% higher order values, outperforming organic in 23 of 25 categories, and more than half of AI-referred sessions land directly on a product page against roughly 20% for organic. Orders from AI-powered search were up about 13x year over year. Google's own documentation adds that clicks from AI Overviews are higher quality, with those users spending more time on site.
John Mueller, reported by Search Engine Journal in June 2026, put it plainly: "Don't block agents. I think that hurdle will be the biggest, for most sites."
My read, and you're welcome to disagree: if you sell something, blocking retrieval bots is self-harm dressed up as content protection. Refuse training if you want, that's a legitimate business decision with a clean switch and no visibility penalty attached. But refusing the bots that put you in answers means refusing the buyers those answers send. For a publisher whose entire product is the words, the math genuinely does flip, and 38,066:1 is why.
robots.txt is a preference, not a gate
Here's the part that undermines both camps. TollBit's H1 2026 publisher logs found roughly 15% of AI scrapes ignored an active robots.txt disallow, with ChatGPT-User at 54%. A `Disallow` line is a request, not a mechanism.
Enforcement lives at the WAF or CDN. Cloudflare's AI Crawl Control is free on every plan, which makes it the obvious starting point for most teams. Their Content Signals Policy adds `search`, `ai-input` and `ai-train` directives to robots.txt, and Cloudflare is upfront that these express preferences rather than enforce anything.
One date for your calendar: from 15 September 2026, Cloudflare blocks training and agent bots by default on ad-monetized pages, with search bots still allowed. If you sit behind Cloudflare and haven't chosen a policy, a policy is about to be chosen for you.
The policy I'd ship
- Allow every search and retrieval bot: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`. No exceptions unless you're a publisher who has done the math above and reached a different answer.
- Treat training bots (`GPTBot`, `ClaudeBot`) as a business decision rather than a technical default, and write down the reasoning somewhere your successor will find it.
- Assume user-initiated agents arrive whatever your robots.txt says, and check that they don't hit a 404 wall or a login redirect on the routes that sell things.
- Server-side render anything you'd be upset about being invisible.
- Move enforcement to the CDN and keep robots.txt as a statement of intent.
- Read your logs. If you can't filter access logs by user agent and keep them long enough to see a trend, start there; our notes on security logging and sensitive data management cover that hygiene.
Then measure, because a policy you can't observe is just a guess with better formatting. That's a topic of its own and we went at it properly in how to measure AI visibility.
None of this is exotic engineering. It's log analysis, a rendering decision, and one file with about a dozen lines in it. The reason it stays broken at so many companies is that it sits between the SEO team, the platform team, and whoever inherited the CDN account, which means it belongs to nobody.
If you'd rather have someone else read the logs, we run a free agent-readiness audit: what agents actually see on your pages, which crawler classes are reaching you, and which routes come back blank.
