Decoy Pages and Paywalls for Bots: Cloudflare's AI Labyrinth and Pay Per Crawl

When Google started answering suspected bots with poisoned SERPs, it used the same idea Cloudflare had put into production in 2025: give an unwanted crawler content that wastes its time instead of blocking it.

Cloudflare sits in front of a large share of the web, so its choices affect anyone who crawls at scale. Since early 2025 it has added decoy pages for misbehaving bots, blocked AI crawlers by default for new customers, and started letting sites charge crawlers per request. This article covers what changed and what it means if you collect web data.

March 2025: AI Labyrinth

In March 2025, Cloudflare introduced AI Labyrinth, an opt-in feature that sends crawlers which ignore "no crawl" rules into a maze of generated pages.

How it works, according to Cloudflare and coverage of the launch:

• When Cloudflare detects suspicious bot activity, it adds hidden links to the site's pages.
• The links lead to pages generated with Workers AI. The content is real, often science related, and unrelated to the site.
• The generated pages carry directives that keep search engines from indexing them.
• A person would not follow hidden links four pages deep into unrelated content, so a visitor that does is almost certainly a bot. Cloudflare uses that behavior to identify and fingerprint bad bots.

Cloudflare acknowledged that bots have started looking for hidden link traps, which limits how well the approach works. The idea matters more than the feature. A bot that is caught receives plausible content that looks valid, and the bot doesn't know it was caught.

July 2025: AI crawlers blocked by default

On July 1, 2025, Cloudflare began blocking AI crawlers by default for new domains, unless the crawler has permission. Existing customers can turn on the same blocking from their dashboard.

At the same time, Cloudflare launched pay per crawl in private beta. A site can set a price, and a crawler that wants the page can pay it. Requests without payment receive HTTP 402 "Payment Required." By August 2025, Cloudflare said its customers were sending more than one billion 402 responses on an average day.

August 2025: the stealth crawling dispute

In August 2025, Cloudflare accused Perplexity of using undeclared crawlers that looked like regular Chrome browsers to reach sites that had blocked its declared crawlers. Cloudflare removed Perplexity from its verified bots and added rules to block the behavior. Perplexity disputed the claims.

The dispute showed how detection has moved beyond user agents. Cloudflare said the undeclared traffic rotated through IP addresses and networks not tied to Perplexity's declared crawlers, even though the requests claimed to be a normal browser. Our explainer on how anti-bot detection works covers these signals.

September 2025: Content Signals

In September 2025, Cloudflare published the Content Signals Policy, which adds three signals to robots.txt:

• search: use for search results
• ai-input: use as input to AI answers, such as grounding
• ai-train: use to train AI models

Each can be set to yes or no. Cloudflare applied search=yes, ai-train=no to more than 3.8 million domains that use its managed robots.txt. Cloudflare says the signals "express preferences; they are not technical countermeasures against scraping," but they make a site's wishes clear and machine readable.

2026: payments for agents and accountability for bots

In 2026, Cloudflare kept moving in the same direction. In September it opened a beta of a Monetization Gateway that lets sites charge AI agents through HTTP 402. In October it announced continuous checks on whether verified bots honor robots.txt and rate limits.

What this means for web data collection

The common thread between Cloudflare's tools and Google's poisoned SERPs is that blocking is no longer the only response to unwanted automation. Sites can now answer with decoy content, a price, or a quietly wrong page. For anyone collecting data, that changes the job:

1. Respect the signals. Read robots.txt, including content signals, and honor them. A crawler that respects a site's rules doesn't end up in a labyrinth.
2. Identify your crawler. A site can choose to allow a declared crawler with a clear user agent and contact details. It can only guess about an anonymous one.
3. Don't follow hidden links. Only follow links a person could see and click. Our guide to avoiding blocks covers honeypot traps.
4. Check the content as well as the status code. A 200 response with unrelated content is a sign you were trapped or poisoned. This is the same lesson as silent scraper failures.
5. Budget for paid access. For some sites, paying per request through HTTP 402 may become the normal way in.

None of this is legal advice, but the trend is clear. Sites now have better tools to decide who gets their content and on what terms, and our guide on whether web scraping is legal explains why those terms matter.

Frequently asked questions

What is Cloudflare AI Labyrinth?

AI Labyrinth is an opt-in Cloudflare feature introduced in March 2025. When it detects a crawler ignoring no-crawl rules, it adds hidden links that lead to generated pages unrelated to the site. Bots that follow them waste resources and reveal themselves, while search engines are told not to index the pages.

Does Cloudflare block AI crawlers by default?

Since July 1, 2025, Cloudflare blocks AI crawlers by default for new domains unless the crawler has permission. Existing customers can turn the blocking on from their dashboard.

What is Cloudflare pay per crawl?

Pay per crawl lets a site set a price for crawler access. Crawlers that don't pay receive an HTTP 402 Payment Required response. It launched in private beta on July 1, 2025.

What are Cloudflare Content Signals?

Content Signals are three robots.txt settings introduced in September 2025: search, ai-input, and ai-train. Each tells crawlers whether the site allows that use of its content. They express preferences and don't technically block access.

How is this related to Google's poisoned SERPs?

Both answer unwanted automated traffic with plausible content instead of a block. AI Labyrinth serves decoy pages to crawlers that ignore rules, and Google's poisoned SERPs serve complete results pages with most of the usual sites swapped out. In both cases the bot may not know it was caught.