The Four Ways AI Systems Use Your Content: Crawling, Training, RAG, and Licensed Retrieval

Ask a publisher whether AI is using their content and the answer is almost always yes. Ask them how, and the room goes quiet. That gap is where most bad access decisions get made.

There is no single thing called “AI using your content.” There are four distinct mechanisms, and each one copies your work at a different moment, for a different reason, under a different set of rules. The reflex response, blocking every bot at the CDN, treats all four as one. It is a blunt instrument because the problem is not blunt.

This post explains each mechanism in plain terms: what it does, what it copies, what you can control, and how value moves or fails to. The closing section takes up the argument that the next competitive fight in AI will not be over models at all.

Crawling, training, retrieval-augmented generation (RAG), and licensed retrieval are the four ways AI systems use publisher content. Crawling is the automated fetch that copies a page onto someone else’s infrastructure. Training is the one-time process that turns copied text into a model’s weights, where it stays permanently and cannot be traced back to a source. RAG is the fetch that happens at answer time, when a system pulls your live page or an indexed copy into the prompt so the model can use and cite it. Licensed retrieval is RAG with permission, defined scope, and a record of use attached. The four are legally and economically distinct, which is why a single block-or-allow setting cannot govern them.

Why one switch can’t govern four uses

Most access policies were written for one bot doing one job: the search crawler that indexed your pages and sent readers back. AI broke that model in two directions at once. The same fetch can now feed a training run you will never see or a cited answer you would be glad to appear in, and the operators’ own documentation distinguishes those cases even when publishers don’t.

OpenAI publishes three separate crawler agents for three separate purposes: GPTBot for training, OAI-SearchBot for search inclusion, and ChatGPT-User for pages a person asks ChatGPT to open. Each is controlled independently in robots.txt, and the documentation says so in as many words. Google goes a step further. Its crawler reference documents that Google-Extended is not a crawler at all. It is a control token applied to content Googlebot already fetched, deciding whether that content may be used for Gemini training and grounding. Disallowing it changes nothing about Google Search.

If the operators themselves separate the mechanisms, a publisher who collapses them back into “block AI” is negotiating with less precision than the other side. Our earlier post on whether publishers should block AI crawlers covers that tradeoff in detail. This one is about what the mechanisms are.

Mechanism 1: Crawling, the fetch

Crawling is the automated visit. A bot requests your URL, your server responds, and a copy of the page now exists somewhere you don’t control. Every downstream use begins here, but crawling on its own decides nothing about what happens next.

What it copies: the page as served, usually text, sometimes images and structured data.

What triggers it: a schedule or a discovery queue. Nobody is asking a question when a crawler arrives.

What you control: robots.txt directives per user agent, rate limits, firewall rules, and bot verification against published IP ranges, since user-agent strings can be spoofed by anyone. Google’s crawler documentation lists both the agent strings and the IP range files for exactly that reason, and OpenAI publishes the same for each of its agents.

How value moves: it mostly doesn’t, not yet. Crawling is a cost event for you and an input event for them. The question that matters is which of the other three mechanisms the crawl feeds. A crawl that feeds a search index sends readers back. A crawl that feeds a training corpus sends nothing back, ever. Same fetch, opposite economics.

The common mistake is judging the fetch instead of the use. Log analysis by user agent tells you who is fetching. Only the operator’s documentation tells you why.

Mechanism 2: Training, the copy that becomes weights

Training is the process that turns a corpus of copied text into a model’s parameters. This is what most people mean when they ask “does AI train on my content,” and the honest answer is that it depends on which agents you allowed, and when.

What it copies: the whole corpus, repeatedly. The US Copyright Office’s report on generative AI training walks through the chain: downloading and curating a dataset makes copies, staging it for a training run makes more, and the resulting weights may retain memorized passages, which is disputed but not dismissible. The report also records Common Crawl telling the Office that its open web crawl supplied roughly 82 percent of the raw tokens behind GPT-3. Open web content is not a side ingredient in training. It is the base.

What triggers it: a training run, which happens rarely and is invisible to you. Nothing in your server logs says “this request is for training.” The only signal you get is the agent name, and only if the operator is honest about which agent does what.

What you control: forward-looking opt-outs. Disallowing GPTBot signals that your content should not be used to train future OpenAI models. Disallowing Google-Extended does the same for future Gemini models. Neither reaches back into a model that already exists. If your pages were crawled before those tokens were introduced, the training mechanism has already run, and the block protects the next model, not the last one.

How value moves: once, diffusely, and in a way that is close to impossible to meter. A corpus of billions of documents cannot attribute a given answer to a given page, so nobody can tell you what your archive contributed to a model. That is why deals for training rights tend to be flat fees, and why those deals leave publishers without usage data when renewal comes around, a problem we covered in What Publishers Give Up in Flat-Fee AI Licensing Deals. Training is the mechanism where “how much was my content worth” is hardest to answer, so it is the one most often paid for with a round number.

Mechanism 3: RAG, the fetch at answer time

Retrieval-augmented generation is the mechanism doing most of the work in AI search today, and the one publishers most often mistake for training.

What’s the difference between AI training and RAG?

Training changes the model. RAG changes the prompt. In training, your content is absorbed into the weights once and cannot be located or removed afterward. In RAG, your content is fetched at the moment a question is asked, placed into the model’s context alongside that question, and used to generate that one answer. The model itself is unchanged. That is why RAG answers can carry citations and training-based answers cannot: the retrieval step knows where the text came from, and the weights never did.

The term comes from a 2020 paper by Patrick Lewis and colleagues, who described pairing a model’s parametric memory, meaning what training put into the weights, with non-parametric memory, an external index the model can query on demand. Every AI search product is a descendant of that idea, whether it retrieves from a private index or from the live web.

What it copies: the Copyright Office describes two RAG patterns. In the first, the operator copies material into a retrieval index ahead of time. In the second, the system fetches from an external source, such as a search engine or your live page, at generation time. Both make reproductions, and the Office singles out RAG-related uses as a particular concern for news media.

What triggers it: a person asking a question. This is where ChatGPT-User’s visits come from, and OpenAI’s documentation notes that because those visits are user-initiated, robots.txt rules may not apply to them.

What you control: OAI-SearchBot governs whether you appear in ChatGPT search answers. Google-Extended governs grounding in Gemini Apps and Vertex AI. Google AI Overviews and AI Mode, on the other hand, draw on the ordinary Googlebot index, so the only way out of them is out of Search entirely. Perplexity and Copilot publish their own agents and controls. The specific list changes often; the durable point is that retrieval controls are separate from training controls, and usually more granular.

How value moves: per use, attributably. Each retrieval is a discrete event tied to a specific page and a specific answer. That makes RAG the only one of the first three mechanisms that can be counted, and counting is the precondition for being paid. Citations, referral clicks, and any future per-use payment all flow through this lane. If your goal is to get cited by AI, this is the mechanism you are optimizing for.

Mechanism 4: Licensed retrieval, RAG with terms attached

Licensed retrieval is technically the same operation as RAG. The system fetches your content at answer time and uses it to ground a response. What changes is everything around the fetch: permission is explicit, scope is defined, attribution is required, and the use is recorded.

A useful public example is Microsoft’s July 2026 agreement with Nine Entertainment. Under it, Copilot can reference the text of Nine’s mastheads beyond the paywalled preview to ground answers, displays snippets and headlines with references, and directs readers to the full story on Nine’s sites. Read that against the RAG description above. The mechanism is identical. The difference is that Nine chose the scope, set the conditions, and has a counterparty accountable for the use.

What it copies: whatever the terms say, and nothing more. Full text behind a paywall is on the table precisely because it is licensed. An unlicensed retrieval agent should never get past the preview.

What triggers it: a person’s question, same as RAG, but inside a relationship rather than a default.

What you control: the terms. This is the first mechanism where control is contractual rather than technical, which means it survives an agent that ignores robots.txt.

How value moves: per use, with a record. Licensed retrieval is where a usage report becomes possible, where a renewal can be argued from data rather than guesswork, and where the economics of AI answers start to resemble the economics of any other distribution channel. It is also the mechanism the AI Content Economy is being built around.

The four mechanisms side by side

Mechanism	When it copies	Who triggers it	What you control	How value flows
Crawling	On fetch	A schedule	robots.txt, rate limits, IP verification	Not yet. Depends on the downstream use
Training	Before a model is built	A training run	Forward-only opt-out tokens (GPTBot, Google-Extended)	Once, diffuse, unmeterable
RAG	At answer time	A person’s question	Search and grounding tokens, separate from training	Per use, attributable, unpaid by default
Licensed retrieval	At answer time	A person’s question	Contract terms	Per use, recorded, compensated

Read down the last column and the case against blanket blocking writes itself. A full block removes you from the two mechanisms that can pay you in order to opt out of the one that mostly can’t.

What this means for your access policy

You don’t need a policy for each bot. You need a policy for each mechanism, and then a map from bots to mechanisms.

Start with your logs. Break out AI traffic by user agent and check it against the operators’ published IP ranges. You will usually find that training crawlers and retrieval agents are different entries, and that their volumes are nothing alike.

Then decide per mechanism. Training is a yes-or-no question about future models. Retrieval is a question about where you want to appear and on what terms. Licensed retrieval is a question about who you want to negotiate with and what you need back, starting with usage reporting.

Finally, treat the retrieval lane as the product. The training lane is a one-time sale you can’t price. The retrieval lane is a recurring, countable flow of your content into answers people are reading right now. That is the lane worth metering, worth pricing, and worth protecting with something sturdier than a robots.txt line.

The next AI war isn’t about models

Here is the argument from the whiteboard, and it follows from everything above.

Model capability is converging. The systems answering questions today are built on similar architectures, trained on overlapping corpora, and separated by margins that narrow with each release. Whoever wins the next phase will not win on parameter count. They will win on what they can retrieve: trusted data, with rights cleared, with provenance attached, at the moment a person asks.

That is a fight over four things publishers already hold. Trusted data is your archive and your newsroom. Rights are yours to license or withhold, mechanism by mechanism. Provenance is your byline, your timestamp, your masthead, the things a retrieval system has to show a reader before the reader will believe the answer. And evidence, the record of what was used and when, is what turns the other three into something you can invoice, defend, or carry into a negotiation.

The training era ran on content that arrived with none of that attached. The retrieval era can’t. An answer that has to cite its source has to know its source, and a system that has to know its source has to deal with whoever owns it. The four mechanisms are not four threats. Three of them are the shape of the next market, and the fourth is the one you can finally opt out of.

Frequently asked questions

Does AI train on my content?

If your pages were openly crawlable before you set training opt-outs, it is likely that some models were trained on them, and no opt-out removes content from a model that already exists. Disallowing GPTBot and Google-Extended in robots.txt signals that future models should not train on your content. Those signals do not affect search inclusion or answer-time retrieval, which are governed separately.

What is RAG in plain terms?

RAG is an AI system looking something up before it answers. Instead of relying only on what it learned during training, the system fetches relevant text at the moment of the question, reads it, and generates an answer from it. Because the fetch happens per question, the system knows where the text came from and can cite it. That is the mechanism behind most AI search citations.

If I block GPTBot, will ChatGPT still cite me?

Yes, as long as OAI-SearchBot is allowed. OpenAI documents the two as independent: GPTBot governs training and OAI-SearchBot governs search inclusion. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Note that OpenAI may reuse one crawl for both purposes when a site allows both agents.

Does blocking Google-Extended remove me from Google AI Overviews?

No. Google-Extended controls whether content Googlebot already crawled may be used for Gemini training and for grounding in Gemini Apps and Vertex AI. It has no effect on inclusion in Google Search, and Google AI Overviews draw on the Search index. The only way to leave AI Overviews is to leave Search, which is a much larger decision.

What’s the difference between RAG and licensed retrieval?

The fetch is the same; the terms are not. RAG in its default form retrieves whatever it can access and owes you nothing beyond a citation. Licensed retrieval retrieves what a contract allows, attributes as the contract requires, and generates a record of use that both sides can see. The record is what makes usage-based payment possible.

Where this leaves publishers

Four mechanisms, four sets of rules, and only one of them, training, is a pure loss to opt out of. Crawling is the input to everything else and should be managed, not blocked wholesale. RAG is where citations and referrals come from today. Licensed retrieval is where the record of use lives, and the record is what gets a publisher paid.

NextNet’s publisher program is built for the last two lanes: managed access that keeps you in AI answers, with the usage record that turns retrieval into a line item instead of a favor. If you want to see what a metered retrieval lane looks like for your titles, start here.