The title asks a question, and an opinion piece owes you its answer early: as an SEO tactic, llms.txt is a dead end. The interesting part is that you don't need my word for it. The web has run this exact experiment several times before, and it kept the receipts. You could read the file's fate in its design from the first week, in the incentives, long before the fetch logs confirmed it.
So this is a walk through how web standards actually live and die. I'll place llms.txt in that history, make the strongest case I can for the other side, and end with dated predictions you can grade later. You should also know my bias. We sell AI visibility measurement, and in a world where a text file moved citations, you'd need us less. Weigh my view with that in mind, then check the evidence, which doesn't care about either of us.
How web standards win
Three standards conquered the publisher web, and they share an anatomy worth looking at before you judge the newcomer.
Robots.txt came first, in 1994, and it solved the consumer's problem rather than the publisher's. Crawlers were hammering servers and operators were furious. A convention that let crawlers behave earned them legal cover and goodwill, and it reduced the load they caused. The consumer had every incentive to fetch and honor the file, which is why the fetching started immediately and never stopped. Publishers adopted it because consumption was guaranteed.
XML sitemaps, in 2005, carried no incentive mystery at all. Google launched the format itself, and the other engines co-signed. The consumer announced "we will read this" before asking anyone to write it, so adoption followed consumption, in the right order. Search Console then let you see the payoff.
Schema.org came in 2011 from a consortium of the consumers themselves: Google, Microsoft, Yahoo and Yandex. They defined what they'd parse and then paid publishers visibly for compliance, with stars, panels and other rich results. You could describe yourself, but only within bounds. Your markup got checked against the page content, and you could watch the reward loop play out in your own listings.
Line the three up and you'll see one consistent anatomy. The standards that win are demanded or guaranteed by consumers first, and you as the publisher can observe the payoff. In the whole history of the web, no publisher-side file has succeeded by being published widely and waiting for readers to take an interest.
How web standards die
The graveyard has an anatomy too, and one grave matters most here.
The keywords meta tag was publisher self-description in its purest form. You told the engine what your page was about, in your own words, where no visitor could see it. It worked briefly. Then exactly the spam you'd predict destroyed it, and every search engineer alive learned that self-declared relevance can't be trusted at web scale. Google confirmed in 2009 that it ignores the tag. The afterlife is the most instructive part of the story: CMS templates dutifully kept filling the tag in for two more decades, consuming effort while doing nothing.
Google Authorship, which ran from 2011 to 2014, adds a second lesson: consumption is what starts a standard, and it's also the only thing that keeps one alive. Even a standard with a committed consumer dies the moment that consumer withdraws. Publishers did everything right, photos in results and all, and the program ended anyway.
Set those two graves beside the three winners and the diagnostic question for any new publisher-side convention writes itself. Who has committed to reading this, and what stops the spam that self-description invites? Keep that question in your head for the rest of this piece. It does all the analytical work this topic needs, and it'll do the same work on the next convention the industry invents.
200+ SaaS teams already track their AI citations.
They know exactly when ChatGPT mentions their brand, and when it stops. Do you?
Where llms.txt sits
Now put the patient on the table. llms.txt is publisher-side self-description, proposed in late 2024. It wears robots.txt's clothes (a plain text file at the root, perfectly respectable) while carrying the keywords meta tag's incentives. The publisher speaks, no consumer has committed to listen, and the content is a self-curated map of what the publisher would like machines to believe matters.
Two years in, the consumption record is thin. Google says plainly that it doesn't use the file, and its search relations team has compared it to the keywords meta tag in as many words. Neither OpenAI nor Anthropic has committed to it. In one r/AISEOTricks thread, a lone commenter cut through the optimism on exactly that point: no major AI crawler currently reads it; it's a proposal. Ahrefs then measured the null directly. Of 137,000 domains that had the file, 97 percent logged zero fetches in a month.
And yet adoption grew fivefold in a year, and the same thread shows you why better than any analyst. One commenter keeps the file because "it doesn't hurt in any way." Another says their coding assistant noticed the file was missing and deployed one unprompted, and a third points to a Lighthouse-adjacent audit that checks for it.
So what's driving adoption is tooling defaults, checklist anxiety and the no-harm rationalization, three forces that work without anyone reading the file. If you remember the keywords meta tag's afterlife, you've seen this pattern, with one difference. That ritual was kept going by templates and audits long after the readers left, and with llms.txt the readers never arrived at all.
The hype cycle, reconstructed
A verdict lands better with the timeline next to it, so let me walk you through how two years of discourse actually went. The proposal appeared in late 2024, elegantly simple, and the early coverage was appropriately modest. People treated it as a docs convention with bigger aspirations.
Through 2025 the AI-SEO content economy discovered it. The file became a listicle staple and an audit checkbox, and a deliverable an agency could ship in an hour, which is a powerful thing for a deliverable to be. Adoption charts went up and to the right. Then the charts got cited as evidence of importance, a substitution nobody flagged.
Measurement arrived in early 2026 with null logs, Google comparing the file to the keywords meta tag, and the 97 percent figure. The discourse split into the two camps that Reddit thread shows in miniature: the no-harm keepers and the logs-don't-lie skeptics. By mid 2026, platform deployments had pushed adoption up another leg while consumption stayed flat, and the decoupling was complete.
I'm reconstructing all this because you can reuse the pattern. In any convention's discourse, watch for the moment adoption statistics start standing in for consumption evidence. That substitution is the tell, and here it happened around the middle of 2025, a full year before anyone widely measured the null.
The trust asymmetry nobody can fix
I want to make the strongest version of the why, because "engines are lazy" misses it. Picture yourself running retrieval for a major engine and weighing whether to honor llms.txt at scale. Every SEO on earth now controls a file that tells your system which pages to prefer, so the spam would be a certainty, complete with its own conference circuit. You'd have to check each file's claims against your own crawl anyway. That means computing the map you already compute, plus a reconciliation step, plus an adversarial-abuse team.
In the best case the file is redundant for you, and in the realistic case it's a new attack surface. Declining to fetch it is the correct engineering decision, the one your own team would make, and that's why the logs are empty everywhere at once.
The same logic explains the one place where consumption is real. llms.txt works exactly where its trust problem disappears, and that's documentation for coding agents. The publisher wants the agent to succeed, and the agent's operator wants accurate docs. Nobody profits from lying, and the consumer is often a tool the publisher's own users run, so the incentives line up with little room for spam, and the fetches are real.
Opinions should show their work, so here's my own updating. When the proposal appeared I filed it as "harmless, plausibly early," the same hedge half the industry made, and in 2025 I'd have written you a softer version of this essay. I now think a standard that spreads while unread is decorative rather than early, and a decoration with maintenance costs is a debt. Google's statements didn't move me there, since self-interest could explain them. What did was the uniformity of the null across every independent log anyone published, while adoption kept growing anyway.
How often does ChatGPT mention your brand?
Most founders have no idea. The answer might surprise you.
The costs nobody itemizes
The no-harm argument deserves a direct answer, because it's the rationalization holding up every thread on this topic, including the one embedded above. So let's itemize what "doesn't hurt" actually spends.
Start with opportunity cost. The hours you spend writing and maintaining the file come out of a budget that could have fixed a rendering bug or reconciled a contradicted price, and that work has measured effects. Then there's false completion. Teams that ship the file report feeling covered on AI visibility, and that felt coverage measurably slows the structural work, because ticking the box dulls the urgency.
Misinformation is the third risk. An unmaintained index at a canonical address is the one artifact here that could actively mislead the rare agent that does read it, and unmaintained is the default fate of files outside your CMS workflow. Normalization is the last one. Every audit that flags a missing llms.txt trains a generation of marketers to treat compliance rituals as optimization, which makes the next decorative standard easier to sell. If you want a way to rank that work against everything else on your list, our guide to prioritizing AEO projects walks through it.
None of these costs is large on its own. Each one is nonzero and recurring, though, and you pay all of them for a file whose measured benefit is zero outside one niche. The whole method of this essay is to keep better books than that. "Doesn't hurt" is only true under accounting that ignores every column except the server bill.
The steelman, priced
The other side deserves its best case. Three events could flip this verdict, and I'll give you my odds on each.
The sitemaps play would flip it fastest: a major engine commits. If OpenAI or Google announced fetch-and-honor, consumption would be guaranteed, and everything I've said about incentives would invert within a quarter. It's the only event that fully revives the file. I'd put it under ten percent in the next two years, because the trust asymmetry is structural and the engines keep saying so out loud.
A slower route runs through agent commerce, if it makes curated indexes economic. As agents start acting on sites rather than just reading them, a machine-readable map of "here is how to interact with us" could earn real consumption, with token economics favoring curated context over crawled sprawl. I find that plausible, maybe twenty percent as a meaningful force by 2028. Notice what it describes, though: a successor convention with verification built in, closer to schema.org than to llms.txt, and probably under a different name and governance.
My bet is on the third, where the docs niche expands until it's the story. Coding-agent consumption grows, adjacent tool ecosystems join in, and llms.txt settles into being real and useful but category-specific, the way RSS lives on in podcasts. It's less a flip than a graceful retirement into the one job the file was ever good at, and I'd call it likelier than not.
None of the three rewards you for writing the file today for SEO reasons, which is the practical point. Watch for announced consumption. Ignore the adoption statistics, including the impressive ones, because we've now seen exactly what adoption without consumption means.
The verdict, graded later
So, helpful standard or dead end? For SEO it's a dead end, and for developer documentation it's a minor living convention. It's also a genuinely useful case study in how to evaluate the next artifact this industry produces, because there will be a next one. The whole evaluation kit fits in two questions: who consumes this, and what stops the lying? Put them to every AI-visibility convention that comes along and you'll skip years of ritual compliance.
If you want the file anyway, the recipe and the honest expectations are one post over. The hedge costs you an hour a quarter, and you should price it as insurance rather than as a tactic. The work that measurably moves what engines say is where it's always been: extraction-ready structure, consistent facts and real corroboration. Pair that with a weekly read on your citation share, so you can check claims like this essay's against your own data instead of anyone's opinion, mine included.
These predictions are dated September 2026, so you can grade them later. No major engine commits to llms.txt consumption within two years. Outside developer documentation, the file's fetch rates stay statistically indistinguishable from zero. And at least one successor convention with consumer-side verification appears and gets further than llms.txt did. If the first prediction fails, I'll write the correction at the same word count this verdict got.
15 hours a week manually. Or 15 minutes with RankControl.
Track citations, monitor competitors, and fix content gaps across every AI search engine. Automatically.
