Why URLs Won’t Be Enough in Agentic AI

For thirty years the web has run on one assumption: if you can point to something, you can get to it. A URL tells a browser where a page lives, the browser fetches it, and a person reads what comes back. Search engines, ad markets, analytics, and the entire economics of publishing were built on top of that loop.

AI agents break the loop. They don’t browse. They retrieve, extract, and act on content for a user who may never see the page it came from. That shift changes what content has to carry with it, and it exposes how little a URL carries on its own.

This post is for the people building and governing that shift: publishers, standards readers, and the engineers deciding what an agent may do with what it finds. It covers how agents find content today, why an address alone can’t support machine-scale use, and which three things have to travel with the content instead.

How do AI agents find content?

AI agents find content by retrieving it, not by browsing it. An agent takes a goal from a user, breaks it into sub-tasks, queries a search index or an API for candidate sources, fetches the pages or documents it selects, extracts the passages it needs, and then cites them or acts on them. The user sees the result, not the pages. Every step in that chain is machine to machine, and the only thing the agent needs from a URL is somewhere to send the request.

Researchers describing the agentic web frame this as a move from human-driven interaction to machine-to-machine interaction, where intent is delegated to an agent that plans and executes on the user’s behalf. The same paper argues that the web’s foundations, including HTTP, HTML semantics, indexing, and search, need to be reconsidered for agents rather than people.

That reconsideration is already underway. It just isn’t happening in one place, and most of it isn’t visible to publishers yet.

Consider what an agent skips. It doesn’t render your layout. It doesn’t see your paywall prompt unless the server enforces one. It doesn’t read your terms of service page, because nothing in the fetch tells it a terms page exists. It pulls the text, keeps what matched the sub-task, and moves to the next source. The four ways AI systems put content to work, from training to real-time answers, are covered in our post on how AI systems use content. Discovery is where all four begin.

What a URL does, and what it doesn’t

A URL is a locator. That’s the entire job. It resolves a name to a host and a path, the server decides what to return, and the exchange is over. Nothing in that transaction says who asked, what they may do with the response, or whether anyone kept a record.

Links locate. They don’t license.

For a person in a browser, the gap never mattered much. A human reader brings context the URL doesn’t: they see the byline, notice the copyright notice, decide whether to subscribe, and use the content in ways a court would recognize as reading. For an agent fetching thousands of pages an hour, none of that context exists unless it’s encoded in something the agent can parse.

Three things are missing from the address, and each one blocks a different part of the agentic web from working.

Terms

A URL carries no statement of what the content may be used for. Training, retrieval, summarization, and search indexing are different uses with different economics, and the address treats them identically. If the terms aren’t attached to the content, an agent can only guess, and most guess in their own favor.

Identity

A URL carries no proof of who is asking. The requesting side identifies itself with a User-Agent string and an IP address, both of which can be set to anything. A server that can’t tell one agent from another can’t apply terms to either of them. Terms without identity are a sign on a door with no lock.

Receipt

A URL carries no record of the exchange. Server logs capture that a request happened, but not what was licensed, to whom, under what terms, or what was owed. Without a receipt there is nothing to settle, nothing to audit, and nothing to reconcile when two parties disagree. The reasons that gap has persisted are laid out in our post on why AI still has no system for paying for content.

Why robots.txt and preference signals don’t close the gap

The obvious response is that the web already has a way to tell crawlers what to do. It does, and the standard that defines it is candid about its limits.

The Robots Exclusion Protocol was formalized as RFC 9309 in 2022. The document states plainly that its rules are not a form of access authorization. A crawler matches its own product token against the file and is asked to honor the result. There is no verification that the crawler is who it says it is, no vocabulary for licensing, and no record of compliance. Robots.txt was designed to keep well-behaved crawlers from overloading servers. It was never designed to govern rights.

The IETF’s newer work goes further and still stops short. The AI Preferences working group is chartered to standardize a vocabulary for expressing preferences about AI use of content and ways to attach those preferences to content. That is real progress on the terms problem. But the charter lists technical enforcement, authenticating or authorizing clients, and auditing as explicitly out of scope. A preference is something a publisher states. It is not something an agent is bound by.

None of this is a criticism of the standards bodies. Scoping is how standards ship. The point is that each of these efforts solves one slice of the problem, and none of them was designed to be the whole answer. Read together, they map the shape of what’s missing.

What has to travel with the content instead

Three layers are taking shape, each in a different standards body, and each answering one of the three gaps above. They weren’t coordinated, but they fit together.

Terms a machine can read

Really Simple Licensing 1.0 was published as a specification in December 2025 with a permanent namespace. An RSL document is an XML file that defines licensing terms for one or more assets: which uses are permitted or prohibited (training, AI input, search, and others), what form payment takes (attribution, subscription, per-use), and optionally a license server the client must obtain a license from. It can be attached through robots.txt, HTTP headers, embedded in pages, or carried inside media files.

RSL is the terms layer. It turns “here is my content” into “here is my content and here is what you may do with it.” That’s a necessary step, and on its own it has the same enforcement gap the AI Preferences charter describes. Terms need someone to bind to.

Identity a server can verify

The IETF’s Web Bot Auth working group is developing a protocol for HTTP message signatures for automated traffic. The idea is simple: an agent signs each outbound request with a private key, includes a header pointing to where its public keys are published, and the receiving server verifies the signature. The draft is explicit that today’s identification methods, meaning IP allowlists, User-Agent strings, and shared API keys, have significant limitations.

Two details in the draft matter for this argument. First, the authors note that the identifying URL on its own carries nothing; it is a claim, not an identity, until the server resolves it and finds a key that verifies the request. Second, the protocol deliberately does not define authorization. It tells a server who is knocking and leaves the decision about whether to let them in to something else, which is exactly where terms and receipts enter.

Provenance that survives the copy

The Coalition for Content Provenance and Authenticity defines Content Credentials, a cryptographically signed manifest that records an asset’s history: where it came from, what edits were made with which tools, and whether AI was involved in creating it. The manifest travels with the asset rather than living at a URL, and the specification allows publishers to add their own assertions, including whether they permit use for AI training.

The explainer is careful to say that Content Credentials make no judgment about whether content is good or true. They establish that a set of facts about the content hasn’t been tampered with. For an agent, that’s the useful part. It can’t evaluate editorial quality, but it can check a signature.

From links to content objects

Put the three layers together and the unit of exchange changes. Instead of an address that resolves to whatever the server returns, the agent receives a content object: the content itself, plus the terms it’s offered under, plus a verifiable record of where it came from. The agent’s own signed identity completes the exchange, and a ledger records it. The anatomy of that object is the subject of our guide to rights-managed content objects, so this post won’t repeat it.

The closest analogy is the barcode. A barcode doesn’t say where a product sits on the shelf. It identifies the product itself, which is why any scanner in any store can price it, count it, and settle it without calling the manufacturer. Content on the agentic web needs the same thing: an identifier bound to the work rather than its location, with the terms and the record attached. The URL keeps its job as the delivery address and stops being asked to do everything else.

This is also what makes per-piece licensing possible. A site-wide robots.txt rule treats every page the same. A content object lets a publisher offer an evergreen explainer under one set of terms and a breaking investigation under another, and lets an agent honor both without a human negotiating either.

Why provenance becomes the trust signal

Agents have a problem human readers don’t. They can’t judge a source by feel. A person weighs the masthead, the writing, the date, and a dozen cues they couldn’t name. An agent has the text and whatever metadata came with it.

What an agent can check is verifiability. Does the content carry a signed manifest that validates? Do the terms resolve to a license the agent can accept? Does the signature match the party the terms name? Each of those is a yes-or-no test, which is the kind of test software is good at.

The direction this points is that agent builders will have reasons to prefer content whose origin can be verified. Verified sources reduce the risk of citing a manipulated page. They make attribution mechanical instead of best-effort. And they give the agent’s operator a defensible answer when someone asks where a claim came from. The Web Bot Auth draft makes a related point from the server side: an unverified request is not evidence of bad faith, but it also isn’t evidence of anything, and policy has to treat it accordingly. Content works the same way in reverse.

Expect provenance to start functioning the way a domain’s reputation did in the search era, as a signal that shapes which sources get pulled into an answer. For publishers, that means the work of attaching terms and credentials to content isn’t only a rights exercise. It’s a visibility exercise.

What publishers can do now

None of the three layers requires abandoning URLs, and none of them requires waiting for the others to finish. Each is useful on its own and more useful together.

Publish machine-readable terms. Add an RSL document to your robots.txt or serve it in HTTP headers, and define usage types by section rather than site-wide. This is the lowest-cost step and the one every other layer depends on.

Prepare to recognize signed agent traffic. Start logging any signature headers you receive, and decide in advance how your server will treat verified, invalid, and unverified requests as distinct outcomes. You don’t need to enforce anything yet to start learning who is asking.

Attach provenance to your highest-value assets first. Original reporting, photography, and data products are where verifiability pays off soonest. Content Credentials tooling exists today for images and video, and the specification covers documents.

Count usage per piece, not per site. A receipt only exists if someone records the transaction at the level of the work. Even before a settlement layer is in place, knowing which content objects are being retrieved and by whom is the data every future licensing conversation will run on.

Keep the URL. It’s still how content gets delivered. The change is additive: you’re giving the address something to carry.

How all of this fits into the broader shift in who pays for content and why is the subject of our AI Content Economy pillar.

Where SAIL fits

Next Net built SAIL, the Standardized Agentic Intelligence Ledger, for the part of this stack none of the standards above are chartered to provide: the transaction and the receipt. SAIL is designed to record a license event between an identified agent and a specific content object, under the publisher’s stated terms, and to give both sides a receipt they can reconcile against. It’s built to work alongside terms expressed in RSL rather than replace them, and it’s meant to sit on top of the identity and provenance layers as they mature.

For publishers, the practical outcome is that content can be offered to agents on stated terms, with usage counted at the level of the individual work, without a bilateral deal for every AI company that shows up. Details are on our publisher page.

Frequently asked questions

How do AI agents find content?

They retrieve it rather than browse it. An agent queries a search index or API for candidate sources, fetches the ones it selects, extracts the relevant passages, and cites or acts on them, all without a person viewing the page. The URL is only the delivery address in that process.

Will URLs go away in agentic AI?

No. URLs remain the mechanism for locating and delivering content. What changes is that the address stops being the unit of exchange. Terms, identity, and provenance have to travel with the content, because the address can’t carry them.

What is the difference between robots.txt and RSL?

Robots.txt, defined in RFC 9309, tells crawlers which paths they’re asked not to fetch, and the standard states its rules are not access authorization. RSL defines machine-readable licensing terms: which uses are permitted, what payment applies, and where to obtain a license. Robots.txt governs crawl access, and RSL governs rights.

How can a website tell whether a request came from an AI agent?

Today, mostly by User-Agent strings and IP ranges, both of which can be spoofed. The IETF Web Bot Auth work proposes that agents cryptographically sign their requests so a server can verify the identity behind them. The protocol identifies the agent; it doesn’t decide what the agent is allowed to do.

What is content provenance, and why does it matter for AI agents?

Provenance is a verifiable record of a piece of content’s origin and history, typically a signed manifest that travels with the asset. It matters because agents can’t judge quality the way readers do, but they can check a signature. Verifiable origin is likely to become one of the signals that determines which sources an agent pulls into an answer.

The address was never the asset

The web got a long way on locators because a person was always on the other end of the link to supply the context a URL left out. Agents don’t supply it. They need the terms, the identity, and the record delivered with the content, in a form software can verify.

The standards for each layer exist or are close. What’s still being built is the layer that ties them into a transaction. If you publish content that AI systems are already retrieving, that’s the conversation worth having now. See how SAIL works for publishers.