Beyond block or allow: How pay-per-crawl is reshaping public data monetization

For most of the web's history, content platforms operated on a simple binary: open or blocked. Then generative AI changed everything.

February 26, 2026

Beyond block or allow: How pay-per-crawl is reshaping public data monetization

For most of the web's history, content platforms operated on a simple binary: open or blocked. Then generative AI changed everything.

• Credit: Alexandra Francis*

Key takeaways

• The traditional open/blocked model for managing bot traffic is broken. AI crawlers have become too sophisticated for organizations to block them at scale, and they extract value without returning it.
• Pay-per-crawl replaces the binary choice with a “yes, if” framework: programmatic, usage-based access to content gated by real-time payment requirements via the HTTP 402 status code.
• Stack Overflow and Cloudflare have co-launched a pay-per-crawl model, offering organizations a new path to public data monetization that complements, not replaces, traditional data licensing.

Why the “open vs. blocked” internet model is failing

For most of the web's history, content platforms operated on a simple binary: open or blocked. Bots that followed the rules, like search engine crawlers and legitimate aggregators, were welcomed. Bots that misbehaved were added to a blocklist. It was an imperfect system, but it was more or less functional.

Then generative AI changed everything. The explosion of LLMs created enormous commercial demand for high-quality training data, and the web became the most convenient source of that data. AI crawlers began hitting content sites at unprecedented scale to extract data for model training. The reciprocal traffic loop that once underpinned the internet's content economy began to collapse.

"With the rise of AI products looking to take data for model training, we found ourselves in a position in the last year or so to revisit that approach," said Janice Manningham, Strategic Product Leader at Stack Overflow, on the Leaders of Code podcast. The old open-or-block framework, she explained, simply wasn't built for this moment: “We needed to protect our data against commercial usage for model training, but still allow access to our community.”

Why blocking AI crawlers isn't enough

At the dawn of the generative AI era, Stack Overflow, like many content platforms, began maintaining blocklists of aggressive AI crawlers. But as Josh Zhang, Site Reliability Engineer at Stack Overflow, explains, this approach quickly hit its limits.

"We're basically just playing whack-a-mole," Zhang said. "There are other tools you can use like fingerprinting and bot scoring, but of course it's an adversarial relationship — so the people writing the bots know what they have to defeat."

The sophistication of modern AI crawlers has escalated well beyond simple curl requests. Today's bots use headless browsers to convincingly mimic human traffic. This means they're not just scraping content, but also consuming ad impressions. Advertisers are paying for traffic from human users, but AI crawlers are fooling the verification systems in place to ensure that. "They're basically eating up ad impressions," Zhang noted, "which is also a really terrible thing to take back to the advertisers themselves."

The result is an arms race that most content teams can't win by playing defense. The blocklist at Stack Overflow grew unwieldy. Scaling the manual, ad hoc identification process would have required a significantly larger team. The team needed a different strategy: Rather than simply reacting to bot traffic, they would redirect it.

What is pay-per-crawl?

Pay-per-crawl is a usage-based content access model in which automated crawlers and AI agents are granted programmatic access to web content only upon fulfilling real-time payment and identity requirements. The model empowers content owners to monetize bot traffic directly without blocking public access or requiring human-negotiated contracts.

The model is distinct from the two dominant alternatives that came before it:

• Robots.txt has long functioned as a handshake agreement between website owners and crawlers. It signals preferences, but it is entirely voluntary: There is no enforcement mechanism and no penalties for noncompliance. AI companies have largely regarded it as optional.
• Paywalls solve the revenue problem, but create an access problem. They are designed for human readers and, by definition, require friction: account creation, credit cards, subscription decisions. This makes them incompatible with programmatic, machine-to-machine content access.

Pay-per-crawl uses the existing HTTP 402 ("Payment Required") status code, a rarely implemented code that has been part of web infrastructure for decades, to communicate access terms directly to the bot in real time. The message is not a "no." It is, as Will Allen, VP at Cloudflare, puts it, a "yes, if."

"You are welcome to come get this if there's some sort of payment that happens in here," Allen explained. "And that payment can happen directly, programmatically, machine to machine."

Why it matters: the AI data economy

AI is projected to add up to $4.4 trillion annually to the global economy, and high-quality, licensed data is fueling that growth. Demand for structured, authoritative training datasets is only accelerating as model developers look to differentiate on data quality.

For content owners, this creates a meaningful opportunity missing from the old open-or-block model. Traffic from AI crawlers represents a real form of commercial interest that, under traditional frameworks, generates cost (server load, ad impression distortion) with no corresponding revenue.

Pay-per-crawl enables content owners to meet that commercial interest where it already is. Rather than waiting for AI companies to initiate formal licensing conversations, organizations can respond directly to bot activity, creating a pull mechanism that surfaces potential partners and generates transactional revenue from crawlers who would otherwise extract data for free.

Benefits of pay-per-crawl for content owners and organizations

Let’s get into the benefits of the pay-per-crawl model for both content owners and organizations seeking to make use of that content.

Revenue from uncompensated traffic. Bot crawlers that used to extract data without payment can now be required to pay for access. Even at a low per-crawl rate, high-volume AI training traffic can represent meaningful public data monetization.

Flexible data access on your terms. Unlike comprehensive data licensing agreements, which typically involve lengthy procurement cycles, broad dataset access, and significant negotiation overhead, pay-per-crawl supports granular, usage-based access. Crawlers pay for what they use when they use it. This opens the door to potential customers who may not be ready for or interested in a full licensing deal.

Reduced uncontrolled scraping. The 402 response itself functions as a signal. When Stack Overflow enabled pay-per-crawl, some bots that had previously received a hard 403 block simply stopped sending traffic after receiving the 402. Message received: "It's almost like they got the message," Zhang said. The 402 communicates intent — “This content has value, and access requires acknowledgment” — without the blunt force of a full block.

A mechanism for surfacing licensing conversations. Not every interaction will result in a machine-to-machine payment. Some will result in something more valuable: a phone call. "Maybe they do some level of machine transactions, but even more so, it's giving them the tools that they need to strike these deals across the board," Allen said. The 402 response functions as an invitation to negotiate.

[...]

Original source