Exclusive.org

Digital ideas, domains and editorial insights

Cloudflare Data Shows Human Traffic Falling Up to 40% as AI Bots Take Over the Web

By

For the first time in the history of the public internet, machines generate more traffic to websites than people do. New data from Cloudflare puts automated requests at 57.5% of all web activity, against 42.5% from human visitors — a crossover that Cloudflare’s own CEO had predicted wouldn’t happen until 2027.

The shift is being driven by agentic AI: software that browses and retrieves web pages on behalf of assistants like ChatGPT and Gemini, often visiting thousands of pages in the time it would take a person to open a handful of browser tabs. Cloudflare’s crawler data shows the composition of that traffic changing too. As of June 2026, 52% of crawler requests exist purely to feed AI training systems, up from 22% just over a year earlier. Pure search indexing, the kind that eventually sends a visitor back to the source, now makes up a shrinking share of the total.

The Cost to Publishers

The practical effect is a growing gap between how much a site is crawled and how much it’s actually visited. Content gets pulled into an AI system’s answer, but the reader never clicks through to the page it came from. Cloudflare reports that publishers in the most heavily crawled categories — including software, IT services, and finance — have seen human traffic fall by as much as 40% in under a year.

The behavioral shift behind that number is stark: Cloudflare estimates that of every hour people spend online researching something, only about 15 minutes is now spent on the open web itself. The rest is absorbed into AI-generated answers that never route a visit back to the original publisher.

Cloudflare’s Response

Cloudflare has started giving publishers more granular tools to manage the shift. Crawlers are now split into three categories — Search, Agent, and Training — so site owners can block or charge each independently rather than treating all bots the same way. New default settings, effective September 15, will block Training and Agent bots on ad-supported pages unless a publisher opts out, a change broad enough to catch even Googlebot. The company has also opened a waitlist for a Monetization Gateway, aimed at letting publishers charge AI companies directly for access to their content.

Earlier steps have already shown measurable effects: after Cloudflare introduced a one-click AI-crawler block last year, more than a million customers enabled it, and traffic from ByteDance’s Bytespider crawler fell more than 71% across Cloudflare’s network.

The open question is whether these tools can restore the old exchange — content for traffic — or whether the web is settling into a new model where content is consumed by machines on behalf of humans, with far less of the reciprocity that funded publishing for the last three decades.