AI Bot Management: Your CDN May Be Overruling Your Robots.txt
Allow or block is no longer enough. Here is why AI bot management needs a real policy, and what I would check on your CDN, robots.txt and GA4 now.
Allow or block is no longer enough. Here is why AI bot management needs a real policy, and what I would check on your CDN, robots.txt and GA4 now.
A recent piece by Chris Green on Search Engine Journal argues that AI bot management can no longer be a simple allow-or-block decision in robots.txt. The real enforcement often happens in the CDN, the firewall or a bot mitigation tool, sometimes without the SEO team knowing. If you care about showing up in AI Overviews, ChatGPT or Perplexity, this is worth checking.
For a long time, bot handling in SEO was fairly simple. Googlebot and Bingbot got access, a few scrapers got blocked, and robots.txt did most of the work. It was not perfect, but it covered what most sites needed.
AI search has broken that model. The visitors hitting your site now include traditional search crawlers, AI training crawlers, retrieval bots, user-initiated fetches and autonomous agents. They do very different jobs. Still, most of us put them all in one bucket called "AI bots" and make one decision for the whole group.
That is where the risk sits. A rule written to stop training scrapers can also cut you out of AI answers.
The key point in the article, and I agree with it, is that blocking a training crawler has very different consequences from blocking a retrieval bot. Roughly, this is how I would split them:
Some AI companies publish separate user agents or tokens for training and for search, for example OpenAI separates GPTBot and OAI-SearchBot, and Google uses Google-Extended for Gemini training. Check each vendor's documentation instead of relying on old lists, because these change.
In my experience, many sites that "blocked AI" in 2023 or 2024 did it with one broad rule. Some of them now wonder why they never get cited in AI answers. That is the first thing I would look at.
Robots.txt is voluntary. Well-behaved crawlers follow it, others ignore it. So the real enforcement happens elsewhere: in the CDN, the web application firewall and dedicated bot mitigation services.
Those layers are usually owned by engineering or security, not by the SEO team. A CDN may have a default "block AI crawlers" toggle switched on. A firewall rule may challenge anything that looks automated. The result can be that your robots.txt says "allow" while the CDN quietly says "no". Nobody in marketing sees it, because there is no error in Search Console and the robots.txt file looks fine.
The opposite also happens. You may think you blocked a training crawler, but if it ignores robots.txt and nothing enforces the rule at the edge, it still gets your content.
This is why I would treat robots.txt as a statement of intent, not as the control itself. The control is wherever requests actually get accepted or rejected. This kind of check is now a standard part of how we run a technical SEO audit.
Agents add another layer. They can browse a site like a real user, run JavaScript and trigger your analytics tags. Over time they become harder to tell apart from human visitors.
Two practical consequences follow. First, if your security setup blocks an agent that is doing a product comparison, you do not just lose a position somewhere. In the article's words: "You could disappear from the answer entirely." For an e-commerce site, that means losing the shortlist, not only a click.
Second, agents that do get through can show up in GA4 as sessions. If your conversion rate or engagement suddenly drops on certain product pages, I would at least consider whether automated traffic is diluting the data before changing bids or landing pages. This is where clean web analytics and tracking setup pays off: segmenting by user agent, hostname and behaviour patterns where possible, and keeping server logs available for comparison.
The article's main argument is to move from block lists to deliberate bot policies: who is accessing the content, why, and what happens if you allow or block them. I think that is the right direction. In practice, this is what I would do:
If you are tracking AI visibility already, for example after the recent model changes in AI Mode we covered in our post on Gemini 4 Hydrogen and AI visibility, bot access is the first thing to rule out when citations drop. There is no point optimising content that the retrieval bot cannot reach.
E-commerce sites, publishers and anyone running a large site behind an enterprise CDN. These are the setups where security defaults are most likely to override SEO intent. Smaller sites on standard hosting are usually less exposed, but a single plugin or CDN toggle can still change everything.
My honest view: most sites do not need a complex framework. They need one documented decision per bot type, one person who owns it, and a log check every few months. That alone puts you ahead of many competitors who still do not know what their CDN is blocking.
AI bot management means deciding which AI crawlers, retrieval bots and agents can access your site, and enforcing that decision consistently across robots.txt, the CDN and the firewall. The goal is to protect content where needed without losing visibility in AI search answers.
Only the ones that choose to respect it. Robots.txt is voluntary, so actual enforcement happens in the CDN, WAF or bot mitigation tools, which can also block bots your robots.txt allows.
For most commercial sites I think that is a reasonable starting point, because retrieval bots affect whether you appear in AI answers while training bots do not directly. Blocking training crawlers is a business and legal decision, so involve those teams.
Source: AI Bot Management: Why SEOs Need More Than Block Lists (Search Engine Journal)