What actually stops an AI crawler

Two things happen in the same month and they look unrelated. A scheduled automation you built to check a supplier’s pricing page starts returning 403 instead of HTML. Separately, a reader mentions that a chatbot summarised the good half of your guide back to them, accurately, without sending them anywhere near your site. Those are the same story arriving from opposite directions, and you are standing on both sides of it.

The useful question is not whether any of this is fair. It is narrower: which controls actually exist, which of them a crawler can simply decline to obey, and what changes for the automations you have already built. This guide covers what stops a crawler, what only asks one nicely, the gap that neither closes, and what to do when the block lands on your own tools. It is written for people who publish something worth reading and run a few automations that read other people’s sites. It is not for anyone running crawling at commercial scale, where the question is contractual and legal rather than technical, and none of this is legal advice.

Asking a crawler to leave was never a mechanism

robots.txt was finally written down as an Internet Standards Track document, RFC 9309, in September 2022 [1]. The document is unusually blunt about what it is not. The rules it carries “are not a form of access authorization”, and the protocol “is not a substitute for valid content security measures” [1]. A crawler that ignores your robots.txt has not broken through anything. It has declined a request that was only ever a request.

That was tolerable for as long as the crawlers that mattered wanted the same thing you did. Search indexing was a trade: you let the bot read the page, the bot sent you readers. Both sides had a reason to keep the etiquette. Remove the return traffic and the etiquette is all that is left, held up by nothing.

Patreon is the tidy illustration of what happens next. In July 2026 it moved from asking AI training bots to stay away to blocking them at the network, using Cloudflare’s AI Crawl Control, and its stated reason was the whole argument in one line: “Consent shouldn’t depend on whether a scraper chooses to behave” [8]. During testing, weekly access attempts from individual AI training crawlers reportedly dropped from thousands to zero, which tells you what those crawlers had been doing with the polite version [8]. Patreon kept allowing bots that index pages and send people back to it [8]. The distinction it drew is the one to copy: not bots versus no bots, but bots that return something versus bots that do not.

Evasion is documented, not hypothetical. On 4 August 2025 Cloudflare published an investigation alleging that Perplexity was “repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity”, using a generic user agent impersonating Chrome on macOS from “multiple IPs not listed in Perplexity’s official IP range”, at 3 to 6 million daily requests next to 20 to 25 million from the declared crawler, across tens of thousands of domains [5]. Whatever you conclude about that specific dispute, the capability is now general knowledge. A directive in a text file is not a defence against anyone who has decided not to read it.

The trade that made crawling tolerable stopped paying

The economics are the reason this keeps escalating, and Cloudflare put numbers on them when it announced on 1 July 2025 that it was “changing the default to block AI crawlers unless they pay creators for their content” [2]. Measuring pages crawled against visitors referred back, it reported that with OpenAI “it’s 750 times more difficult to get traffic than it was with the Google of old”, and with Anthropic “it’s 30,000 times more difficult” [2].

Treat those as one company’s measurement of its own network rather than a law of nature. The direction still matters more than the decimal place. If a thousand pages of your work are read for every visitor who arrives, the exchange that justified open crawling has quietly become a donation. That is the calculation behind every platform that has hardened its rules in the last two years, and it is one you can run on your own site with numbers you already have.

The controls, in the order they actually bind

There are three layers, and confusing them is where most people go wrong.

The first is robots.txt, which states who may crawl. It is a preference, honoured by the operators who choose to honour it [1].

The second layer is newer and answers a different question: not who may read the page, but what they may do with it afterwards. Cloudflare’s Content Signals Policy, published on 24 September 2025, defines three machine-readable signals to sit alongside your existing rules: search, for “building a search index and providing search results”; ai-input, for “inputting content into one or more AI models”, including retrieval-augmented generation and grounding; and ai-train, for “training or fine-tuning AI models” [4]. Cloudflare applied Content-Signal: search=yes, ai-train=no to more than 3.8 million domains using its managed robots.txt, and deliberately left the middle signal unset, on the grounds that “we don’t know their preference with respect to that signal, and we don’t want to guess” [4]. That middle signal is the one worth thinking about hardest, because it governs the case where a model reads your page live to answer somebody’s question today, which is where most of your traffic loss actually comes from.

The third layer is the only one that is a control rather than a statement. Cloudflare’s AI Crawl Control, formerly AI Audit, lets you “set allow or block rules for individual crawlers”, gives you insight into “how AI crawlers are interacting with your pages”, and tracks “which crawlers follow your directives” so you can “create enforcement rules” [3]. It is available on all plans [3]. The same product carries the interesting future option, currently in private beta: allowing AI crawlers “to access content by paying per crawl” [3]. Other CDNs and hosts offer their own equivalents; the layer matters more than the vendor.

Run them in that order. Say what you want in robots.txt and in the content signals, because the operators who honour directives read them and because a stated preference is the thing anyone later argues from. Then enforce the part you actually care about at the network, because that is the only layer that does not depend on the reader’s manners.

The fetch a person asked for goes through anyway

Here is the gap that no amount of blocking closes, and it is written plainly in the vendors’ own documentation.

OpenAI runs separate bots for separate jobs: GPTBot crawls “content that may be used in training our generative AI foundation models”, OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features”, and ChatGPT-User handles “certain user actions in ChatGPT and Custom GPTs” [6]. About that last one, the documentation says: “Because these actions are initiated by a user, robots.txt rules may not apply” [6]. Anthropic splits the same three ways: ClaudeBot “collecting web content” for its models, Claude-SearchBot which “navigates the web to improve search result quality”, and Claude-User, which accesses websites “when individuals ask questions to Claude” [7]. Its help page says Anthropic’s bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”, notes that Claude-User “allows site owners to control which sites can be accessed through these user-initiated requests”, and publishes its crawler IP ranges as a JSON file [7].

So the practical answer to “can I stop AI reading my site” is: you can stop the bulk training crawl, you can usually stop the search index, and you cannot stop a person pasting your URL into a chat window and getting a summary. Block all three categories and you have opted out of appearing in the AI search results that are gradually replacing the ones you used to rank in, while the summarising still happens on request. That is a real trade with no comfortable option, and it is worth making deliberately rather than by leaving the defaults alone.

Notice that this is also the door your own research tools walk through. When you paste a link into Claude or ChatGPT, or run a Perplexity query, you are using the user-triggered fetcher, which is the category least likely to be blocked and most likely to be identified as yours.

When the block lands on your side

Assume the sites you pull from will get stricter, on their own schedule, without telling you. The failure mode is quiet: an n8n or Zapier HTTP step that used to return a page now returns 403 or a challenge page, and the automation downstream keeps running on stale or empty data until somebody notices the report is wrong. Anything scheduled and unattended deserves a check that the fetch actually returned what it used to, not merely that the run finished.

The durable fix is boring and it is the same one every time. Prefer the official route: an API, an RSS or JSON feed, an export, a partner relationship, a paid data licence. Those persist because somebody at the other end has agreed to them, which is exactly what a scraped page never has. Where no official route exists, keep the scraped source but treat it as weather rather than infrastructure: one source among several, with the automation degrading visibly instead of silently when it goes.

Do not route around a block. The Perplexity episode is the case study in what that costs, because the accusation that stuck was not that the company crawled, it was that it hid [5]. You are a smaller target with a more fragile reputation, and rotating user agents to get past a refusal is the kind of thing that reads badly in an email from someone’s lawyer. If the block was a mistake, the operator’s contact address is a faster fix than the workaround.

The optimistic reading of paid access is worth keeping in view. If pay-per-crawl style arrangements move out of private beta and become normal [3], the outcome is a web where access is identified and priced rather than fought over, and small publishers get a mechanism they have never had. That is a possible future, not a current product, and nothing about this year’s roadmap should depend on it.

What still goes wrong

Blocking is a moving target maintained by other people. Crawler identification depends on user agents and published IP ranges that anyone can decline to use honestly, which is the entire substance of the Perplexity dispute [5], and detection that leans on behaviour rather than declarations will occasionally catch something you wanted, such as a link preview, an accessibility tool, or a search engine you actually rely on. Nothing here is set once. The list of bots to consider gets longer every quarter, and the categories they fall into keep being redrawn by the vendors themselves.

The vocabulary is also still being negotiated. Content signals are a policy layer built on top of robots.txt by one vendor with a large footprint, published in September 2025 [4], not a settled internet standard, and the categories underneath them are younger still. The split between reading a page to index it and reading it to answer a question live [4] barely existed as a named distinction before this wave of tools arrived. Anything you write today may need rewriting in a syntax that has not been agreed yet, which is an argument for keeping your stated preference short and your enforced rule specific.

And enforcement does not reach the copies. Whatever your site says now, material published openly before you tightened anything may already sit in training sets, in caches, in archives and in other people’s reposts, and no robots.txt written in 2026 reaches backwards. Everything above changes what happens next. It does not change what already happened, and if the answer you need is about work already taken, that is a legal question for a lawyer rather than a configuration question for a CDN.