When your AI agent hits a paywall

You ask an agent what five sites say about a topic. Ninety seconds later you get a tidy answer and five links. Four of those pages were actually fetched. The fifth returned a subscribe button and nothing else, or a login wall, or a payment-required response. The agent did not mention it. It wrote a paragraph about that source anyway, assembled from the headline, the URL slug, and whatever it already knew about the publication.

That is the real problem, and it is not that agents get blocked. Being blocked is normal and often correct; the site owner is entitled to it. The problem is that a blocked fetch and a successful fetch land in the model’s context looking similar enough that the model treats them the same, and the answer that reaches you carries no mark of the difference. This guide is for people who point agents at the web and act on what comes back: research assistants, monitoring runs, anything with a URL in it. If you operate a crawler at scale, or you are a publisher deciding what to charge, go to the vendor documentation directly. This is the other half of the problem, the half you sit on.

The failure that costs you is the one that returns 200

A page can fail to reach your agent in three ways, and only one of them looks like failure.

The first is a clean refusal. The server returns 401, 402 or 403, the fetch tool reports an error, and something in the chain knows the request did not work. Anthropic’s web fetch tool returns url_not_accessible when the underlying HTTP request fails, alongside a short list of other named errors including url_not_allowed for URLs blocked by domain filtering or by robots.txt, and too_many_requests for rate limits [6]. Named errors are the good case. You can see them, count them, and decide what to do.

The second is an empty success. The page returns 200 with a body, but the body is a subscribe prompt, a consent interstitial, or a shell that fills in with JavaScript after load. Anthropic’s documentation states that the web fetch tool “currently does not support websites dynamically rendered with JavaScript” [6]. To the model this is not an error. It is a short page. A short page about a topic it knows something about is exactly the condition under which a language model produces confident filler.

The third is the overlay paywall, where the content is technically present. Columbia Journalism Review’s analysis of AI browsers, published in October 2025, found that publications using a client-side overlay paywall are exposed precisely because “the text loads on the page but is hidden behind a pop-up that asks a user to subscribe or log in” [8]. A fetch tool reading the HTML gets the article. A human gets a modal asking for a subscription. Your agent may be reading things you did not intend to take, and reporting nothing unusual, because from its position nothing unusual happened.

The practical consequence is that your coverage of a topic is uneven in a way the output does not show. You cannot fix that by writing a better prompt about accuracy. You fix it by making the fetch layer report.

Four responses that mean four different things

Treating every failed fetch as “site unavailable” throws away the one piece of information that tells you what to do next.

RFC 9110 defines 401 as a request that “has not been applied because it lacks valid authentication credentials for the target resource” [4]. That is a login wall, and it is the one case where you may already have the answer: you might hold a paid subscription to that publication. It is a credentials question, not a blocking question.

The same document defines 403 as a server that “understood the request but refuses to fulfill it” [4]. No retry, no session, no header will change that. Record the source as unread and move on, or find the material elsewhere.

Then there is 402. RFC 9110’s entire definition is that the code “is reserved for future use” [4]. The future arrived without waiting for the specification. Under Cloudflare’s pay per crawl, a crawler either presents payment intent in request headers and gets a 200, or “receive[s] an HTTP 402 Payment Required response with pricing” [2]. A 402 is not a refusal. It is a quote.

Rate limiting is the fourth case and the only one where waiting is the correct response. Anthropic’s tool surfaces it separately as too_many_requests [6]. If your workflow retries a 403 on a schedule, you are generating traffic that will never succeed, from an agent whose name is being logged.

Your agent has a name, and site owners now sort by it

Agents do not arrive anonymously. OpenAI publishes its user agents and what each is for: GPTBot for training foundation models, OAI-SearchBot for surfacing sites in ChatGPT search, OAI-AdsBot for checking submitted ads, and ChatGPT-User for user-initiated actions inside ChatGPT [5]. The distinctions carry consequences. OpenAI’s documentation says that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links” [5]. And for the one that matters most to you, the documentation is direct: because ChatGPT-User actions are initiated by a user, “robots.txt rules may not apply” [5].

Site owners have started sorting the same way. Cloudflare’s July 2026 update splits AI traffic into three behaviours rather than one bucket: Search, meaning anything that collects or indexes content to answer questions about it later; Agent, meaning “automated behavior that is acting, usually in real time, on a person’s behalf, to get something done right now”; and Training [1]. From 15 September 2026, new domains onboarding to Cloudflare get defaults where Training and Agent traffic is blocked on pages that display ads, while Search stays allowed [1]. Website owners who do not want the new defaults can mark that in their Security settings any time leading up to 15 September [1].

Read the category names again, because that is the durable part. Your research agent is not a search crawler. It sends no referral traffic and shows nobody an ad. Cloudflare puts chat fetch bots and browser-use agents in the Agent category by name [1], which is the category the defaults are moving against. Assume the population of pages an unauthenticated agent can read gets smaller each year, and design as if it will.

Paying for access is becoming a real option, mostly not for you

The mechanics of paid crawling are now specified rather than theoretical. Cloudflare’s documentation shows the exchange: the first request returns HTTP/2 402 with a header such as crawler-price: USD 0.01; the crawler retries with either crawler-exact-price or crawler-max-price, the latter being “the maximum price the AI crawler is configured to pay for access on any content”; a successful response then carries crawler-charged showing the exact amount billed [3]. Cloudflare acts as merchant of record and provides the underlying technical infrastructure, and the feature was still in closed beta as of its late July 2026 documentation [2].

One detail worth holding onto: if a site blocks a crawler through the WAF or Bot Management, those rules “override pay per crawl’s ‘charge’ feature, and the blocked crawler will not have access to the zone” [2]. Willingness to pay does not create a right to read. A block is still a block.

For a solo operator this is not usually a lever you pull yourself, because the account relationship is between the crawl operator and the network. It is still worth pricing, because it tells you what a research habit is worth. At the documented example price of one cent per request [3], a workflow fetching a few dozen pages a day is a rounding error, and one fetching thousands is a line item.

Getting through is a decision, not a workaround

Some agents do get past paywalls, and the fact that yours can is not the same as permission. CJR’s testing found that OpenAI’s Atlas and Perplexity’s Comet retrieved a subscriber-exclusive MIT Technology Review article that the standard ChatGPT and Perplexity interfaces could not [8].

The reverse behaviour is the one to worry about. CJR reported that Atlas “seems to avoid reading content from media companies that are currently suing OpenAI”, and that when it was asked to summarise an article from PCMag anyway, the agent “produced a composite summary, drawing on tweets about the article, syndicated versions, citations in other outlets, and related coverage across the Web” [8]. The output of that run looks like sourced research and is a reconstruction from third parties.

Vendor policy is not silent here. Anthropic’s Usage Policy, effective 15 September 2025, prohibits actions or behaviours that “circumvent the guardrails or terms of other platforms or services” [7]. That covers instructing a model to defeat an access control on a site you do not control, whatever the technical route.

Three routes stay clean. Use your own credentials on a service whose terms permit automated access, which is a question to answer by reading those terms rather than by testing what works. Use an API, feed or licensed dataset the publisher offers on purpose, which also gives you structured output instead of scraped HTML. Or accept the gap and say so. A guide built on four sources you could read beats one built on five where the fifth was invented from a headline.

Make the workflow report what it could not read

The fix is boring and it is at the fetch layer, not the prompt layer.

Count attempts against successes. If a run was told to check eight URLs and returned six bodies, the answer should say six. In a chat interface, ask for the list of URLs the tool actually retrieved and compare it to the list you gave. In Zapier, n8n or Make, configure the HTTP step so a non-2xx response fails the run loudly instead of passing an error body downstream as if it were content.

Check length, not just status. Set a floor, something like a few hundred words for an article you expected to be long, and treat anything under it as a suspected wall rather than a short page. That single check catches most consent screens and subscribe stubs, which are short by nature.

Then make the model’s instructions match. Tell it in writing to list any URL it could not read and to leave that source out of the analysis rather than infer its contents. Models follow this when asked plainly, and it converts a silent gap into a visible one.

What still goes wrong

Detection stays heuristic. A 200 response with 300 words might be a paywall stub or a genuinely short post, and no rule separates them reliably. The length floor catches the common cases and will occasionally flag something real, which is the correct direction to be wrong in.

Bot identity is mid-transition. User agents are self-declared, and vendors publish IP ranges so sites can verify them, as OpenAI does for each of its crawlers [5], but the verification burden sits with the site rather than with you, and different sites will resolve the same agent differently on the same day. CJR notes the harder version of this: AI browsers such as Comet and Atlas “appear in site logs as normal Chrome sessions” [8]. You cannot predict from your side what a given site will do with a given agent, so the answer is to observe and record rather than to model it in advance.

The legal picture is unsettled in the space between “this site blocked me” and “this site’s terms forbid it”. Vendor policy gives you a clear floor: Anthropic’s Usage Policy rules out circumventing the terms of other services [7]. Where terms are ambiguous, or where a subscription you pay for is silent on automated reading, that is a judgement call and occasionally a question for a lawyer, not a prompt engineering problem. When the stakes are commercial, ask the publisher. Several will say yes, and a written yes is worth more than a technique that works this month.

Prompts from this guide

unreachable-source-report

Research {topic} using the URLs below. Fetch each one.

Rules:
1. Before the analysis, output a two-line report: "Read: N of M" and a
   list of any URL you could not retrieve, with the reason (blocked,
   login required, payment required, empty page, timeout).
2. If a page returned fewer than {min_words} words of body text, treat
   it as unread and list it. Do not summarise a subscribe prompt.
3. Do not describe the contents of a source you did not read. Do not
   substitute coverage of that source from other sites.
4. Every claim in the analysis must be traceable to a page you listed
   as read.

URLs:
{url_list}