The Web’s Broken Bargain: How AI Ended the Deal Between Publishers and Platforms

For the last three decades, the web functioned according to a relatively simple agreement: you published something, others could link to you, index you, or recommend you, and in return you received traffic. Google crawled your pages because it would then send visitors your way. Facebook distributed your content because, at least in theory, it could bring you an audience. The publisher, author, blogger, or company agreed to be part of that ecosystem because there was a clear benefit: visibility, readers, conversation, advertising revenue, subscribers or customers.

That agreement is being pulled apart. AI doesn’t organize the web for us to visit; it consumes it to provide answers on our behalf. In doing so, it no longer necessarily displays a list of links, but rather a summary. It no longer invites the user to go to the source but instead attempts to fulfill their intent right within the interface itself. The economic unit of the web is shifting from the page visited to the response generated. And when that happens, everything else changes: SEO, advertising, e-commerce, media, brands, reputation, copyright, and even the very idea of publishing.

Cloudflare’s pay-per-crawl is the first serious sign that the web is beginning to defend itself against a new form of extraction. Cloudflare offers content owners the ability to allow, block, or charge AI crawlers, even reviving the old HTTP 402 code, “Payment Required”, so that a crawler seeking to access a page is presented with a set price. In its developer documentation, it even explains how a crawler can receive a “402 Payment Required” response with a price for access and return with the necessary headers to pay. In other words: content is no longer a free-for-all by default.

The reason is obvious. If a traditional search engine indexed your site and sent you traffic, while asymmetrical, the relationship made a certain amount of sense. If an AI system crawls your content, summarizes it, uses it to respond to the user, and doesn’t send anyone your way, that logic disappears. What remains is simple appropriation: I take your work, transform it into a response within my product, monetize the user’s attention or engagement, and you’re left with no visitors, no data, no revenue, and, often, no recognition whatsoever.

The data is beginning to confirm this. The Pew Research Center’s analysis shows that Google users who encountered an AI-generated summary clicked on traditional results in just 8% of visits, compared to 15% without the summary; links within the summary itself received clicks in only 1% of cases. Another study on Google AI Overviews and Wikipedia estimated an approximate 15% reduction in daily traffic to articles exposed to these summaries. And a more recent study on AI Overviews observed that responses were triggered much more frequently for queries phrased as questions — those where the user might be satisfied without visiting any sources at all.

The problem isn’t limited to Google: the trend points toward a web increasingly mediated by agents. OpenAI introduced the ChatGPT agent as a system capable of interacting with the web, using browsers, connectors and tools to complete tasks. Previously, it had introduced Operator, an agent with its own browser capable of clicking, typing, and scrolling through pages. If a user stops browsing and begins delegating tasks to an agent, the website no longer receives a person but instead a machine that extracts information, compares options or performs an action. For many websites, that visitor will no longer be a reader but an algorithmic intermediary.

Publishers are already feeling the impact. The Reuters Institute warns that search engines are becoming answer engines, with content served in chat windows, highlighting the risk that referral traffic will dry up. The Guardian explains how changes by Meta and Google are forcing digital publishers to reinvent models that relied on external platforms, with shortfalls in indirect revenue and Google Zero strategies aimed at building direct relationships with the audience.

The uncomfortable but unavoidable conclusion is that if platforms stop sending traffic, they lose much of the legitimacy they once had to access content for free. For years, publishers accepted a dangerous dependency because there was a return. Now, if that return disappears, the contract must be renegotiated. It’s not enough to cite sources merely for show, offer crumbs of traffic or promise exposure in a response that satisfies the user before they even reach the link.

I should point out here that I have no dog in this fight: I’ve never depended on advertising or traffic; I’m privileged to write about whatever I want without having to explain myself to anyone and without depending on my stuff being read. I write for myself, prepare topics for my students, and to conduct research — with no other agenda than that. Your comments help me understand the weaknesses in my arguments and the reactions my writing generates; they help me prepare for the discussions to come… but I don’t make a living from them, nor from my traffic.

But for the vast majority of sites, the web is becoming a free source of content for proprietary systems that will eventually replace the web itself. What seems to be becoming increasingly clear is that if content aggregators and search engines want to use content, they’re going to have to pay for it. If they want to summarize it, they’re going to have to compensate those who produce it. And if they don’t want to do that, publishers will have to start blocking them. The web cannot survive if it becomes a vast plantation of raw material for other people’s business models.

—

This post was previously published on Enrique Dans’ blog.

***