Someone runs Lighthouse on your site, or an SEO tool flags it, and there's a line about llms.txt. Then someone in Slack asks whether you need one. Before anyone argues about that, it helps to know what the file is.
An llms.txt file is a Markdown file at the root of a website, at /llms.txt, that gives AI tools a short summary of the site and a list of links to its most useful pages. Jeremy Howard of Answer.AI proposed it on September 3, 2024. It's a proposal, not a standard any AI company has adopted as an input, and that distinction explains most of the confusion around it.
What an llms.txt file is for
The idea is simple. A website built for people is full of navigation, scripts and layout that a language model has to wade through. The llms.txt proposal suggests adding one plain file that says, in effect, here's what this site is and here are the pages worth reading.
In the proposal's own words, "Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links." That makes it an index for AI tools that are looking something up on your site right now, like a coding assistant reading your API docs. Training was never the point. Howard said as much himself in a reply on X in March 2026: llms.txt is for agents, not training.
The proposal has since grown. Version 2 is dated August 10, 2026, and in it Howard says he updated it "based on what I learned from two years of adoption: thousands of sites publish an llms.txt file, documentation platforms generate one automatically, and Chrome's Lighthouse audits sites for one."
That automatic part matters. The proposal's integrations list names docs platforms like Mintlify and GitBook, the Yoast SEO and AIOSEO plugins for WordPress, and Wix as tools that generate the file. Plenty of sites have one without anyone on the team deciding to make it.
What the format looks like
The format is ordinary Markdown with a fixed order. Here's the spec's format section:
In practice, a file has four parts:
- An H1 with the name of the site or project. It's the only required section.
- A blockquote with a short summary of what the site is.
- Optional paragraphs or lists with more detail.
- H2 sections holding lists of links, each written as name, optionally followed by a colon and a note. An H2 called "Optional" marks links an AI tool can skip when it's short on space.
Vercel's file is a clean real example. Here are its opening lines, as served on September 26, 2026:
# Vercel
> Vercel is a cloud platform for building, deploying, and scaling web applications and AI workloads.
Use this index to find machine-readable documentation and platform resources. Follow the linked indexes when you need individual pages.
## When to use Vercel
- Deploy and scale web applications, APIs, and AI workloads from a Git repository or the Vercel CLI.
There's also a companion convention, llms-full.txt, which puts the full text of the linked pages into one file instead of just linking to them. Docs platforms often publish both.
Know exactly what AI says about your competitors.
RankControl's Recon Agent monitors competitor citations across ChatGPT, Perplexity, Claude, Gemini, Grok, Google AI Mode, and Google AI Overviews. See where they show up and you don't.
What we found in 40 real files
On September 26, we requested /llms.txt from 40 well-known software and developer-tool domains, a mix of marketing sites and docs subdomains, and read what came back.
| What we checked | Result |
|---|---|
| Served a real llms.txt file | 32 of 40 |
| No file (a 404 or an HTML error page) | 6 of 40 |
| Couldn't tell (blocked or broken) | 2 of 40 |
| Median file size | About 18 KB |
| Had both the H1 and the blockquote summary | 24 of 32 |
| Had an "Optional" section | 11 of 32 |
| Also served llms-full.txt | 15 of 40 |
That 80% hit rate is not a web-wide number. We picked the exact kind of company where the file caught on first, developer tools with big documentation sites. Across the wider web, our post on llms.txt adoption vs crawler behavior has the broader figures.
The files themselves split into two kinds. Most were index files, a tidy summary plus link lists, the way the spec describes. A few were instruction files. Render's opens by telling agents to "route according to your task" and points them to a separate page of procedures for agents. We'd expect more of those as companies start writing for AI agents that take actions on a site, alongside the ones that only read it.
The spec is loose in practice too. Twilio's file was 2.3 MB with no H1 and more than 9,000 links, which is closer to a site dump than a summary. Notion's skipped the one required section, the H1. And the AI labs were inconsistent about their own sites: Anthropic's developer docs serve one, while anthropic.com returned a 404.
How llms.txt differs from robots.txt and sitemap.xml
The three files get mixed up because they all live at the root of a site. They do different jobs.
| File | What it does | Who documents reading it |
|---|---|---|
| robots.txt | Tells crawlers which parts of the site they may access | Search engines and major AI companies, including OpenAI for its crawlers |
| sitemap.xml | Lists the pages you want search engines to find and index | Google, Bing and other search engines |
| llms.txt | Summarizes the site and links to the pages worth reading, in Markdown | No major AI search engine |
The proposal puts it plainly: "robots.txt and llms.txt have different purposes." A robots.txt block is a rule that crawlers follow. An llms.txt file is a suggestion that a tool may or may not read. It also says a sitemap "isn't a substitute for llms.txt" because a sitemap lists every indexable page, not a short curated set.
If you're setting crawler rules for AI, robots.txt is the file that matters. Our AI crawler access statistics show how large sites handle GPTBot and OAI-SearchBot there.
Who reads llms.txt
This is where most of the arguments start, so here's what each company has said on the record.
Google has been the most direct. Its guidance on generative AI features was last updated July 10, 2026. It lists llms.txt among things you can ignore and says: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."
Back in April 2025, Google's John Mueller compared the file to the old keywords meta tag in a Reddit reply, adding that none of the AI services had said they were using it.
OpenAI's crawler documentation names robots.txt as the way to control its crawlers and doesn't mention reading other sites' llms.txt files. OpenAI does publish one for its own developer documentation. We found no statement from Anthropic or Perplexity documenting that they read the file either.
The one Google product that does look for it is Lighthouse. Version 13.3.0, released May 7, 2026, added an agentic browsing category to its default audits, and that category checks for an llms.txt file. Howard noted soon after that validation was coming to Chrome's developer tools too.
llms.txt validation will be supported in Chrome devtools :) https://t.co/vpgjq5Z4iv
Jeremy Howard@jeremyphowardMay 19, 2026
So the short version is that tools for building and auditing websites check for llms.txt, while AI search engines haven't committed to using it. That gap is why people on Reddit keep asking the same question. In one r/localseo thread asking whether llms.txt matters for SEO and AI, most replies said no, and one pointed out that nobody could say for sure.
Is llms.txt actually important for SEO and AI recognition, or will Google ignore it?
I’ve been seeing a lot of talk about adding an llms.txt file to websites to help AI models crawl and understand content. Some people say it's the “next robots.txt,” while others claim Google and most search engines don’t even use it. So I’m...
26 content formats. Published on your domain. Matched to your brand.
Guides, comparisons, listicles, case studies, and more. RankControl generates content that gets cited by ChatGPT, Perplexity, Claude, Gemini, Grok, Google AI Mode, and Google AI Overviews.
Should your site have one?
If you have developer documentation, turn on the llms.txt your docs platform can generate. Docs platforms build it so current documentation is easy to pull into an AI assistant, and on most of them it's a setting, not a project.
For a marketing site, it's optional. It won't change how Google ranks or cites you, by Google's own statement, and no AI search engine has said it will. If you want one anyway, keep it short, accurate and in the spec's order, and budget an hour. Our guide on how to write an llms.txt file walks through it, and our post on why Google says llms.txt won't help citations covers the reasoning in more depth.
What does move AI answers is the pages themselves: clear pricing, help articles and comparison pages, crawlable in plain HTML, with robots.txt letting the right crawlers in.
Doing it yourself or handing it off
Writing an llms.txt file is an hour's job. Finding out what AI engines read and cite on your site is the ongoing part.
You can check your server logs for AI crawlers and ask ChatGPT and Perplexity your buyers' questions each month. Or RankControl can do that for you. Our analytics log AI crawler visits for each page, and our AI visibility tracking checks up to 50 buyer questions every week across seven AI engines and records which of your pages each answer cited. For a quick first look, run the free AI visibility check.
Your competitors are getting cited by AI. You're not.
Every day without citation tracking is a day your competitors pull ahead in ChatGPT, Perplexity, and Claude.
