Google's AI Is Killing the Web. Now The Web Is Fighting Back

Article Tools

Contact The Editor Articles By This Author

Google became the king of internet search and created a free cornucopia of knowledge by striking a clever bargain with anyone who made a website: Let the company's crawler index your pages and Google would send human readers back.

That trade looks very different today thanks to Google's strategy of using its search infrastructure to strengthen its artificial-intelligence products.

The technique, which has largely flown under the radar, gives the company an unfair advantage over other AI firms while also accelerating the rise of automated traffic on the internet at the expense of humans.

Google is now meeting some resistance from a plucky British regulator and a crusading industry player, which may finally bring some much-needed change.

Googlebot, the firm's automated crawler, has for many years scoured the Internet and downloaded web pages to feed its massive index so that the freshest and most relevant pages could appear on Google's search results.

More recently, though, the line between content collected for search and content used to generate AI answers has become blurred, as the bot is used to help feed both search and Google's flagship AI model, Gemini.

Google's web crawler sees 3.2 times more of the web than OpenAI's version and 4.8 times more than Microsoft's, according to Matthew Prince, chief executive of web security company Cloudflare.

Prince's firm is a traffic gatekeeper for about a fifth of the web, handling traffic and blocking attacks from bad actors, and he's become a vocal critic of Google for bundling its search and scraping systems. He says the strategy makes it difficult for publishers to block AI training without losing the search traffic they rely on.

Dead internet theory

That advantage also has broader consequences. If AI increasingly reads, summarises and rewrites the web without sending readers back to the pages created by humans, the economic incentive to produce original information starts to fade.

AI agents generated more than 57 per cent of web traffic in 2026 versus about 42 per cent for humans, according to Cloudflare, marking the first time in history that machine activity surpassed humans on the Internet.

Prince sees that divergence getting more extreme in the next few years, a trend that echoes the so-called dead internet theory, the idea that machines increasingly create and consume online content. "Humans will be a rounding error on the internet versus what their agents and their bots are doing," he tells me.

Agents are still largely the preserve of software developers and companies, but Mark Zuckerberg said last week that Meta Platforms would soon launch a consumer version for Facebook Messenger, WhatsApp and Instagram "that just works out of the box and is easy enough for billions of people" to use.

There's nothing wrong with bots doing things that are useful, but the internet will become vapid, less original and ultimately less useful if human contribution is no longer rewarded and peters out.

That's what Google's new search system threatens, as people create content only for machines to copy and condense into AI answers and generating economic rewards that never reach the original authors.

Watchdog demands opt-out choice

Google could have addressed that by letting websites opt out of having their data used for AI while still appearing in search results. The company considered giving them the choice in April 2024, just before it launched AI Overviews, then decided not to because it was "evolving into a space for monetisation", according to internal slides that were disclosed as part of a US antitrust court case.

The company has since faced a reckoning on that decision. Prince says Cloudflare gave Google an ultimatum: From September 15, the infrastructure firm will block mixed-purpose crawlers for its customers who are ad-supported — and do so by default. Google could lose access to millions of websites using Cloudflare.

Further pressure came from the UK's antitrust watchdog. In June, the Competition and Markets Authority ordered Google to give website owners a clear choice to block their content from being used for AI products while remaining in search results.

As part of the order, Google can't punish those websites by pushing them lower down in the rankings.

Prince told me there were "rumblings" that Google wouldn't just follow that standard for the UK but apply it globally. A Google spokesperson confirmed to me that the company was indeed testing a new setting that lets websites opt out of its AI answers without hurting their search rankings, and that it will roll that out worldwide once UK testing is done.

That marks a heartening shift from the status quo, where the world's biggest tech firms play by their own rules.

But the company's crawlers remain technically bundled. The same bot still collects a site's content for both search and AI, and websites have to trust Google to respect their wishes. Fully separating Google's crawler into two — indexing for search, and AI scraping — would be a better remedy.

Websites could then block AI scraping themselves and at their own door rather than relying on Google's word — which, as the internal slides from 2024 showed, tends to bend when revenue is at stake.

It would also beat the cruder alternatives: a blanket ban on grabbing data for AI training or a vague demand that AI companies pay publishers.

Google still controls 90 per cent of the search market, making it a de facto gateway to the web, but its exploitative ambitions are becoming more problematic in the age of AI. With any luck, the company's practices will continue to change for the better.

Parmy Olson is a Bloomberg Opinion columnist covering technology. She previously reported for the Wall Street Journal and Forbes and is the author of "We Are Anonymous."