In the early web, navigation was direct. A link written by a person led to a page, read by a person. Today, automated systems account for 57.5% of global HTTP requests to web content, according to Cloudflare, a figure that arrived more than a year ahead of the company's own forecast.
A metric from Cloudflare measures how often AI platforms crawl a website against how often they send visitors back to it. Anthropic's Claude peaked at over 600,000:1 in February 2025, falling to below 2,000:1 by June 2026. OpenAI's ChatGPT held a comparatively stable range of 800 to 1,800:1 throughout. The gap reflects more than interface differences. A portion of crawling is not tied to user interactions at all. Crawls carried out to train or retrain models generate no referrals by design, since the content is absorbed into the model rather than surfaced to a user. Claude is also heavily used in agentic workflows where the model retrieves content without surfacing links to the user. A ChatGPT browser user, by contrast, is more likely to click through to a source. The interface through which we choose to interact with AI may determine not just how we experience it, but what the open web receives in return.
The contrast is sharpest against traditional search. Google averaged approximately 5:1 in recent weeks, a profile where users still navigate to the source.
The Wikimedia Foundation already reported in early 2025 a 50% rise in multimedia bandwidth since January 2024, attributing it to automated scrapers. Roughly 65% of its most resource-intensive traffic came from bots. Publishers are responding. An increasing number of websites now include a robots.txt file that limits or specifies how automated bots are permitted to interact with their content.