Cloudflare Builds Toll Booths for AI Highway Robbery

Cloudflare has launched “Pay per Crawl,” a private beta marketplace that allows website owners to charge AI companies for scraping their content.

The platform lets publishers either set micropayments per bot visit, grant free access, or block crawlers entirely — a fantasy land of control in a world where bots already take what they want.

This marks Cloudflare’s latest attempt to insert itself between desperate publishers and freeloading AI companies by promising a revenue model where none currently exists.

New websites created on Cloudflare will now automatically block AI crawlers unless explicitly granted permission, part of the company’s push for a “consent-first” web scraping future.

Major publishers like TIME, The Atlantic, and Conde Nast have joined Cloudflare’s default-blocking crusade, eager to stop feeding AI bots for free.

The move comes as news sites suffer dwindling traffic from Google Search and brace for the rise of AI chatbots that bypass websites entirely.

Cloudflare says Google scrapes sites 14 times for every referral, OpenAI does it 17,000 times, and Anthropic hits 73,000 — painting a bleak picture for publishers hoping for traffic karma.

The marketplace requires both AI companies and publishers to be Cloudflare users, with Cloudflare playing the middleman in setting crawl prices and dishing out earnings.

Despite its pay-per-crawl hype, the system doesn’t use crypto or stablecoins, missing a perfect opportunity to make the whole idea even more complicated.

Cloudflare admits the true value of the marketplace might only show up in a future where AI agents act like personal assistants — and presumably ask nicely before looting your content.

Still, without strong incentives, it’s unclear why AI companies would pay for something they’re already getting for free — other than to help Cloudflare fulfill its “benevolent gatekeeper” fantasy.