The Crawl-to-Referral Ratio: The One Number Every Publisher Should Put in Front of Leadership

On September 15, 2026, Cloudflare flipped a default that touches a large slice of the web. Training and Agent crawlers are now blocked on ad-supported pages for new domains and for every site on the free tier, unless the owner went in and chose otherwise. Somewhere in your organization, someone with a budget is going to ask what that means for you.

The instinct is to answer with a policy: block everything, or allow everything, or pick a side in the licensing debate. That is the wrong first move. The right first move is a number that describes the trade you’re already making, whether anyone approved it or not.

This post is about that number. You’ll learn what the crawl-to-referral ratio is, where the public benchmarks come from, how to build yours from your own logs, and how to put it on one slide so the people who set policy can make a decision they’ll be able to defend.

The crawl-to-referral ratio is the number of HTML page requests an AI platform’s crawlers make to your site, divided by the number of human visits that same platform sends back, over the same period. A ratio of 5:1 means the platform fetched five pages for every visitor it delivered. A ratio of 5,000:1 means it fetched five thousand. The ratio is always normalized to a single referral, and it is calculated per platform, not for AI as a whole.

Why this is the number, not “AI traffic”

Most publisher dashboards already have an AI traffic line. It counts bot requests, or it counts referral sessions from ChatGPT, Gemini, Perplexity, Copilot, and Google AI Overviews. Either figure on its own is scenery. It fills the slide without telling you whether the relationship is working.

The ratio puts the two sides of the trade in one cell. Crawl volume is the cost side: origin requests, bandwidth, and content leaving your building. Referrals are the payment side: humans who arrive, see ads, subscribe, or convert. Crawls are what you give; referrals are what you get.

Search worked for thirty years because that trade was cheap. Cloudflare’s 2025 Radar Year in Review puts Google’s crawl-to-refer ratio in the low single digits for most of the year, briefly climbing toward 30:1 during a heavy crawl period and settling back near 3:1. DuckDuckGo sat below 1:1 for three quarters. A search engine that fetches your page three times and sends you one reader is a partner.

The AI platforms run on a different order of magnitude. In the same review, Anthropic’s ratio typically ranged from roughly 25,000:1 to 100,000:1 across the back half of 2025, with erratic spikes far higher in the first half when referral traffic was sparse. OpenAI’s ratio was spiky and reached into the thousands at its peak. Perplexity was the lowest of the AI platforms, generally under 200:1 from September onward.

Those are network-wide averages across millions of sites. Your own number will differ, and that is the point. The public benchmarks tell you the shape of the market. Your number tells you what to do about it.

How Cloudflare calculates it, and where it can mislead you

Cloudflare introduced the metric on Radar in July 2025 and published the crawl-to-refer ratio methodology alongside it. The calculation is simple enough to reproduce. Count every request from user agents associated with a platform where the response was HTML. Count every HTML request where the Referer header contained a hostname associated with that same platform. Divide the first by the second.

Two details in that method matter for how you read the result.

First, all of a platform’s user agents roll up together. OpenAI’s training crawler, its search indexer, and the bot that fetches a page when a ChatGPT user asks a question are three different agents with three different purposes, but Radar aggregates them under one platform. When you build your own version, keep them separate. The purpose of the crawl is the whole story, and we’ll come back to that.

Second, the referral count only sees web referrals. Cloudflare notes in the methodology that traffic from Claude’s native app does not carry a Referer header, and the same is likely true for other native apps. Every visitor who arrives from a mobile app or a desktop client is invisible to the denominator. That means published ratios may overstate how lopsided the trade is, and Cloudflare says plainly that it can’t tell by how much.

The metric is also volatile. Cloudflare describes day-to-day swings driven by crawl bursts and referral lulls. Report a rolling window, never a single day, and report the direction of travel rather than a decimal.

How to build your own ratio from your logs

You don’t need Cloudflare to calculate this. You need server logs or CDN logs, a list of crawler user agents, and a spreadsheet. Here is the sequence.

1. Pick a window

Use 28 days. It smooths weekly cycles and is long enough to catch a training crawl that runs in bursts. Line the crawl window and the referral window up exactly.

2. Count crawls per agent, HTML only

Filter log lines to requests where the response content type is text/html and the status is 200. Group by user agent string. Images, scripts, and 404s inflate the numerator without describing content extraction. Keep each platform’s agents on separate rows: the training bot, the search bot, and the user-triggered fetch bot each get their own line. The AI Crawler Directory post in this series lists the agent strings by platform and purpose.

3. Count referrals per platform

Filter to HTML requests from human sessions where the Referer hostname belongs to an AI platform. Your analytics tool can do this as a referral source report if your log access is limited. Map each hostname to its platform.

4. Divide, per platform and per purpose

Crawls divided by referrals, normalized to one referral. Compute it for the platform as a whole so you can compare against Cloudflare’s published figures, then compute it again with only the search-purpose crawler in the numerator. The second number is the one that describes whether the platform’s search product is a fair partner.

5. Add the citation column

Referrals undercount value for the reason above: app users don’t send a Referer, and many answers cite you without anyone clicking. Put a third column next to crawls and referrals that records how often the platform’s answers name you. That column doesn’t come from your logs. It comes from prompting the engines and recording what they say, which is what an AI visibility platform does for you across ChatGPT, Gemini, Perplexity, Copilot, and Google AI Overviews.

Reading the number: four patterns and what each one asks you to do

Once you have a per-platform table, every row lands in one of four patterns. Each one points at a different decision.

High crawl, low referral

This is the training profile, and it’s where most of the drama lives. The platform is reading everything and sending almost nobody back. Your content is being absorbed into a model, and you’re not a destination in the output. This is the row where a block, a license, or a paywall for crawlers is a legitimate conversation, and the ratio is the evidence you bring to it.

High crawl, high referral

A search-shaped relationship. The platform indexes you heavily and sends people. The ratio may still be high in absolute terms, but if referrals are growing, this row is working. Protect it. Blocking here costs you readers to make a point.

Low crawl, low referral

The platform barely knows you exist. Before you spend a meeting on it, check whether you’ve blocked its search crawler by accident. Cloudflare’s 2025 data found AI crawlers were the most frequently fully disallowed user agents in robots.txt files, and a blanket disallow written two years ago can be quietly keeping you out of a search product that would send traffic.

Low crawl, meaningful referral

Rare and worth a closer look. It usually means the platform is fetching pages at answer time rather than indexing in advance, and the referrals are real. This row is a signal to make those pages easy to fetch, easy to parse, and current.

Notice what the four patterns have in common. The question is never whether AI is good or bad for publishers. The question is which platform, doing which job, at what exchange rate. Being crawled makes you part of the scenery inside the answer. Being referred makes you the place the reader goes next. The ratio tells you which one you are, per platform.

What changed on September 15, and why the ratio matters more now

Cloudflare’s new AI traffic options sort bots into three jobs: Search, which indexes you to answer questions later; Agent, which fetches a page on a person’s behalf right now; and Training, which absorbs your content into a model. As of September 15, Training and Agent are blocked by default on pages that display ads for new domains and for free-tier customers. Search stays allowed.

There is a second change that catches publishers off guard. Multi-purpose crawlers are now judged by their most restrictive declared behavior. A crawler that both indexes for search and collects for training is treated as a training crawler wherever training is blocked. Cloudflare names Googlebot, Applebot, and Bingbot in that category. If you turned on the training block last year without reading the fine print, you may have just cut off crawlers that were feeding your best referral rows.

This is exactly why the ratio has to be split by crawl purpose. A default that blocks Training and Agent while allowing Search is a bet that Search is where the referrals come from. Your logs can confirm or refute that bet for your site specifically. Take the confirmed version to leadership; don’t take the assumption.

Cloudflare’s own data supports the direction of the bet at the network level. Training crawling reached seven to eight times the volume of search crawling at peak in 2025, and user-action crawling grew more than fifteenfold over the year. Training is where the volume is, and the referrals aren’t coming from it. What the network data can’t tell you is whether your Search row is healthy enough to justify keeping the door open. Only your ratio can.

If you want the full decision framework, the Block AI Crawlers post in this series walks through it. This post is about the measurement that decision should rest on.

The one-slide version for leadership

Executives don’t need the methodology. They need one slide they can read in thirty seconds and one recommendation per row. Here is the layout that works.

A table with one row per platform and four columns: crawls in the last 28 days, referrals in the same window, the ratio, and the direction versus the prior 28 days. Below it, a single line that names the platform with the worst ratio and the platform with the best referral growth. That’s the whole slide.

Then three recommendations, each tied to a row. Keep Search open where the referral column is growing. Consider a block or a licensing conversation where the training row is extracting at thousands to one. Investigate the rows where you appear blind, because the most likely cause is your own robots.txt.

Two rules for the slide. Never report a single day, and never show a decimal. The public benchmarks move by orders of magnitude between weeks. A ratio of 48,213:1 invites an argument about the number. A ratio in the tens of thousands, up from last month, invites a decision.

Add the citation column if you have it. A platform that crawls you heavily, refers you rarely, but names your brand in a large share of relevant answers is a different conversation than one that does none of those things. Without that column, leadership is deciding on cost data alone.

Where NextNet fits

Your logs give you the crawl side and the web-referral side. They can’t see what the engines say about you, and they can’t see the app users who arrive without a Referer. NextNet’s platform closes that gap by prompting ChatGPT, Gemini, Perplexity, Copilot, and Google AI Overviews across the questions your audience asks and recording where you’re cited, where you’re recommended, and where a competitor is standing in your place. You can see the full measurement set on our features page.

For publishers specifically, that citation data sits next to the crawl-to-referral ratio as the value column the ratio is missing. If you’re weighing the SAIL framework, licensing terms, or a crawler policy, our publisher program is built for that decision. And if you want a fast read on where your site stands today, the report card is the shortest path.

Frequently asked questions

What is a good crawl-to-referral ratio for a publisher?

Low single digits is the search benchmark, and anything under about 10:1 for a search-purpose crawler is a healthy partnership by 2025 standards. Ratios in the hundreds are common for AI search products and may still be worth keeping open if referrals are growing. Ratios in the thousands or higher are the training profile, where the trade is one-sided and a policy decision is warranted.

Does the crawl-to-referral ratio include Google AI Overviews?

Not cleanly. Googlebot crawls for both search indexing and AI training, and a visit from an AI Overview link carries a Google referrer that looks like ordinary search. The ratio for Google therefore blends classic search with AI, which is one reason to pair it with citation tracking that measures AI Overviews directly.

Why does Cloudflare’s ratio for a platform differ from mine?

Cloudflare’s figure is a network-wide average across millions of sites, aggregated by platform rather than by crawler purpose, and it only counts web referrals. Your figure is one site, can be split by purpose, and reflects your audience’s mix of app and web usage. Differences of an order of magnitude are normal. Compare the direction of travel, not the absolute value.

Should I block AI crawlers if my ratio is high?

A high ratio on a training crawler is a reason to consider a block, a license, or a charge. A high ratio on a search crawler with growing referrals is a reason to leave it open. Because Cloudflare now treats multi-purpose crawlers by their most restrictive behavior, blocking training can also block search from the same operator, so check which purposes each crawler declares before you act.

How often should I recalculate the ratio?

Monthly, on a rolling 28-day window, with the prior window alongside for direction. Crawl volumes change with model releases and product launches, and referral volumes change with how each engine displays links. A quarterly number is too stale to act on; a daily number is too noisy to trust.

The number that ends the argument

Every publisher meeting about AI eventually splits into two camps: the people who want to block everything and the people who are afraid of disappearing. Both camps are guessing. The crawl-to-referral ratio, split by platform and by purpose, replaces the guess with the exchange rate you’re already accepting.

Build it from your logs this month. Put it on one slide. Add the citation column so the value side is as visible as the cost side. Then let the row decide the policy, not the room.

If you’d like to see the citation side of that table for your site, request your report card.