AI Feed vs llms.txt: What Actually Helps AI Engines Understand Your Website?

"AI feed" covers at least four different things, from OpenAI's product feed to Microsoft's NLWeb. We checked which ones an engine documents reading, compared them with llms.txt, and tested what shopping answers cite.

RankControl11 min read
AI Feed vs llms.txt: What Actually Helps AI Engines Understand Your Website?

Open Allbirds' llms.txt and you won't find a summary of the site. You'll find instructions. The file tells any AI agent shopping on someone's behalf to "highly recommend" that the user install Shopify's Shop skill, so the agent can search, compare and buy directly. Kith's and Jimmy Fairly's files say the same thing, word for word except the store name, which suggests Shopify generates it.

So even llms.txt is being used as a signpost to something else, a live catalog and checkout that agents can work with. That's the real question behind "AI feed vs llms.txt." An AI feed is anything that hands an engine structured, current data it has said it will use. llms.txt is a text file that, at best, points to one.

"AI feed" means at least four different things

Nobody has standardized the term. Nearly nobody searches it either, at about 90 US searches a month against 5,400 for llms.txt. When people say "AI feed," they usually mean one of these:

MechanismWhat it isWho reads itEvidence it's read
OpenAI product feed (Agentic Commerce Protocol)A catalog file or API feed of your productsChatGPT's shopping resultsStrong: OpenAI documents it
Google Merchant Center feedYour product catalog for GoogleGoogle Shopping, with insights now covering AI Mode and GeminiStrong for Shopping, inferred for AI answers
Microsoft NLWebA live endpoint that answers questions over your schema.org and RSS dataAgents that call it, over MCPDocumented protocol, little live adoption
Markdown on requestThe same page served as clean markdownCoding agents and AI tools that ask for itDocumented by Cloudflare, Vercel, Framer and others
llms.txtA static text file describing your siteNo engine documents reading itWeak

The two product feeds only matter if you sell physical products. Markdown on request matters to almost everyone, and llms.txt sits at the bottom of the evidence column for a reason.

llms.txt, briefly

We've covered llms.txt at length, so here's the short version. No major engine documents reading third-party llms.txt files for search or citations, and Google's own AI search guide from May 2026 called the file unnecessary for its generative AI features. The detail is in our posts on what Google has said about it and what it does for AI citations.

The spec itself has grown up. Version 2 of the proposal, last modified on August 10, 2026, opens by saying agents now use websites "constantly" and that "Today it is routine."

The llms.txt v2 specification page by Jeremy Howard, showing the published and modified dates and the background section

Even developers who work with agents every day are lukewarm on the file. Lee Robinson, a well-known developer educator, said last year that he hasn't found llms.txt helpful himself. He mostly copies pages as markdown or uses an MCP server that indexes docs, and he'd count it a win if llms.txt pushes docs sites to add that kind of "copy as markdown" button.

llms.txt is good for, well, LLMs. I haven't found helpful myself. Instead, I'm usually doing: 1. Copy page as markdown (majority) 2. Using an MCP like Context7 (indexes docs sites) But if llms.txt gets docs sites to add copy as markdown, it's a win for me.

Lee Robinson@leerobAug 31, 2025

OpenAI's product feed, up close

This is the strongest example of a real AI feed. OpenAI's Agentic Commerce Protocol is an open standard built with Stripe. It lets ChatGPT "ingest structured catalog data, understand merchant inventory, and surface relevant products in context."

OpenAI developer docs page for the Agentic Commerce Protocol, describing it as the infrastructure between merchants and shoppers in ChatGPT

The feed has nine required fields:

FieldWhat it holds
item_idYour unique ID for the product
titleThe product name
descriptionThe product description
urlThe product page on your site
brandThe brand name
seller_nameWho sells it
image_urlThe main product image
availabilityWhether it's in stock
priceThe current price

You can send it as JSON Lines or as a CSV in Google Merchant Center's format, which means many stores can reuse the feed they already maintain for Google. OpenAI recommends a full file upload once a day, or API updates through the day.

The catch is access. OpenAI's getting-started guide says onboarding "is currently available to approved partners," and you apply through a form. So for most stores in September 2026, the AI feed with the strongest documentation is one you have to ask to join.

RANKCONTROL

15 hours a week manually. Or 15 minutes with RankControl.

Track citations, monitor competitors, and fix content gaps across every AI search engine. Automatically.

What shopping answers cite today

We asked four shopping questions through OpenAI's API with web search on, and through Perplexity's API, including "best running shoes for flat feet under $150" and "best office chair for lower back pain under $500."

The two engines behaved very differently. OpenAI's model cited brand and merchant product pages in two of the four answers. For the running shoes, six of its seven citations went straight to brands like New Balance, ASICS and Saucony. Perplexity cited zero merchant product pages across all four answers. It went to review sites and roundups every time.

One caveat matters here. Both tests used general web search through the APIs, not ChatGPT's shopping cards, which are what the product feed powers. So this shows how each engine's web answers behave, and it says nothing about what the feed pipeline would show.

It does show why product pages still matter even without a feed. When OpenAI's model searched, it opened specific product pages, like Branch's office chair page, and cited them directly. Shopify found something similar in its own data: more than half of AI-referred sessions start on a product page, against about 20% for organic search, per a Shopify analysis published in May.

Small stores feel this gap most. In another r/ecommerce thread, the owner of a two-year-old camping gear shop described a competitor with fewer reviews showing up in ChatGPT's recommendations while their own store didn't, around the same time their orders slowed. There's no feed toggle that fixes that on its own, which is why the product pages and the roundups that mention you still carry the weight.

r/ecommerce· u/IlllllIdea· Sep 3, 2026

Has anyone figured out how much ChatGPT actually affects where people shop?

Run a small camping gear store, about two years in, just me and one part timer.Orders have been noticeably slower the past few months. Nothing disastrous, but enough that I keep checking the numbers and wondering if something has changed th...

↑ 9 upvotes18 comments
Via Reddit

Google Merchant Center and AI Mode

Google hasn't launched a separate AI feed. Merchant Center remains the only way to hand Google structured product data, and Google's Merchant Center updates now report on product performance in AI Mode, AI Overviews and the Gemini app.

We didn't find a Google statement that says, in so many words, "AI Mode shopping answers are built from your Merchant Center feed." The connection is strong circumstantial evidence rather than a documented rule. If you sell products, keeping that feed accurate was already worth doing for Shopping, so the AI angle is a bonus.

NLWeb and markdown on request

NLWeb is the odd one out. Microsoft built it as an open project that turns a site's existing schema.org and RSS data into a live ask endpoint, and "every NLWeb endpoint is also an MCP server," in Microsoft's words. It isn't a file you publish. It's a service you run.

Adoption is thin. None of the five big retail sites we checked exposed an /ask or /mcp endpoint at the root, though a custom path is possible. Treat NLWeb as something to watch, not something to build this quarter.

Markdown on request is the quiet winner. Cloudflare's Markdown for Agents returns a markdown version of any page when a client asks for text/markdown, and you can switch it on in its dashboard. Vercel serves any docs page as markdown if you add .md to the URL, and Framer does something similar for optimized pages, which we covered in our Framer publishing guide. Unlike llms.txt, this changes what an agent actually receives when it fetches your page.

RANKCONTROL

AI search traffic grew 835% this year. Is your content ready?

RankControl generates 26 content formats optimized for ChatGPT, Claude, and Perplexity. Published on your domain, matched to your brand.

The feeds you already have

Before building anything new, check what your site already publishes. Every site with a sitemap already hands crawlers a list of URLs and last-modified dates, which is the oldest machine-readable feed there is. RSS and Atom feeds do the same for posts, and NLWeb builds directly on top of them.

Shopify stores have one more. Every storefront exposes a /products.json endpoint by default, and Allbirds' returned about 215 KB of structured product data when we checked, with titles, handles, descriptions and prices for each variant. Price trackers and aggregators have read these for years. AI shopping agents can read them the same way, and you didn't have to build anything.

None of these guarantee that an engine will use your data. They do mean the building blocks exist, and the job is usually keeping them accurate rather than adding a new file.

The options side by side

llms.txtOpenAI product feedMerchant CenterNLWebMarkdown pages
Setup effortAbout 20 minutesA day or more, plus approvalYou may already have itA service to runA setting on some hosts
Who can use itAnyoneApproved merchantsAnyone selling productsAnyone with schema or RSSAnyone whose host supports it
Engine says it reads itNoYes, for ChatGPT shoppingYes, for Google ShoppingAgents that call itTools that request markdown
Keeps prices freshNoYesYesYesYes, if the page is current

Stale data is the problem feeds solve

The best argument for a feed isn't discovery. It's freshness. In r/ecommerce this September, a store owner described ChatGPT recommending a bundle they'd stopped selling, at its old $149 price, after they'd raised it to $189 in January. Perplexity gave a different price, and another engine skipped them and named two competitors.

r/ecommerce· u/Currency-2· Sep 22, 2026

chatgpt is recommending a product we discontinued over a year ago

customer messaged us last week asking about a bundle we don't even carry anymore. turned out they got the idea from asking chatgpt for recommendations in our niche. it still had our old price listed too, $149, even though we bumped it to $1...

↑ 6 upvotes19 comments
Via Reddit

A feed that updates daily fixes that on the surfaces it feeds. For web answers, the fix one commenter suggested is simpler: find the old page the engine is quoting, then remove it or redirect it. An llms.txt file can't do either job.

Which one to do first

It depends on what you sell.

If you run a store, start with Merchant Center, since it's the feed you probably already have. Keep prices and availability accurate there, apply for OpenAI's product feed program, and make sure your product pages load their prices without JavaScript. On Shopify, check what your auto-generated llms.txt says, since it's already pointing agents to Shopify's checkout.

If you sell software, you don't have a catalog, so product feeds don't apply. Spend the hour on your pages instead: let AI search crawlers in, put plans and prices in plain HTML, and turn on markdown versions if your host or docs platform offers them. Add an llms.txt if you like. It takes 20 minutes and it won't hurt, and our 12-site llms.txt test shows what to expect from it.

Either way, don't guess whether the bots are reading you. Check your server logs, or use a tool that counts crawler visits. In the logs, look for OAI-SearchBot, PerplexityBot and ChatGPT-User fetching your product, pricing and docs pages, since those are the requests that feed answers.

Requests for /llms.txt are worth a glance too, but check who made them. One SaaS founder who went through their logs found that most requests for the file came from their own checks and from site scanners, with a single fetch from an AI tool in about seven weeks.

Where RankControl fits

You can check all of this yourself: read your logs for AI crawler visits, ask the engines your buyers' questions every month, and note which pages they cite. For stores, our ecommerce AEO guide covers the product-page side.

Or RankControl can take the measuring. Its analytics counts AI crawler visits and AI-referred visits, so you can see whether the bots are reading your pages at all. AI visibility tracking checks up to 50 buyer questions every week on seven engines and shows which sources each engine quoted.

For a quick look first, run a free check of your brand in AI answers.

Built by the team that got cited in 48 hours.

Content generation, backlink building, AI visibility tracking, and Google rankings. One platform, zero guesswork.

Show me the platform →One platform · 7 AI agents

Frequently Asked Questions

There's no single standard. The term usually means one of four things: a product feed submitted to an engine, like OpenAI's product feed for ChatGPT or a Google Merchant Center feed, a queryable endpoint like Microsoft's NLWeb, or markdown versions of your pages served to AI agents.

OpenAI doesn't document reading third-party llms.txt files for search or citations, and neither does any other major engine. Google's May 2026 AI search guide called the file unnecessary for its AI features. It's a small courtesy file, not a ranking input.

OpenAI accepts structured product feeds through its Agentic Commerce Protocol, with nine required fields such as title, price, availability and URL, in JSON Lines or a Google Merchant style CSV. As of September 2026, onboarding is open to approved partners, who apply through a form.

Usually not. Product feeds are built for catalogs of physical products. A software company gets more from clean, crawlable pages, markdown versions of key docs if its platform supports them, and letting AI search crawlers in.

No. llms.txt is a static text file describing a site. NLWeb, from Microsoft, is a live endpoint that answers natural-language questions over a site's schema.org and RSS data, and every NLWeb instance also works as an MCP server.

RANKCONTROL

Ready to rank on Google and get AI citations?

Content that ranks on Google and gets cited by AI search engines. Published on your domain. Citations tracked weekly.

Related Articles

AEO Vs GEO: What Is The Difference?

Nov 2, 2026 · 8 min read

AEO Vs GEO: What Is The Difference?

AEO vs GEO in plain terms: answer engine optimization aims to be the answer, generative engine optimization aims to be cited inside one. We checked how ChatGPT and Google AI Mode define them, what Google says, and how US search demand for each term changed over the past year.

AEO Best Practices In 2026: How To Win Answers Without Chasing Outdated Tactics

Sep 30, 2026 · 9 min read

AEO Best Practices In 2026: How To Win Answers Without Chasing Outdated Tactics

Half the AEO advice circulating today died in the last four model updates. The tactic graveyard, the practices that survived everything, and the filter for what's next.

LLM SEO Vs GEO Vs AEO Vs AIO: What These Terms Actually Mean

Sep 23, 2026 · 9 min read

LLM SEO Vs GEO Vs AEO Vs AIO: What These Terms Actually Mean

The disambiguation the industry owes you: five overlapping acronyms, one practice, two genuinely different meanings of AIO, and which words to use when.

Is SEO Dead Now With AI?

Nov 1, 2026 · 9 min read

Is SEO Dead Now With AI?

Is SEO dead now that AI answers so many searches? We asked ChatGPT and Google AI Mode 10 versions of the question on October 19, 2026, checked where their answers came from, and lined them up with Google's own guidance and the latest click data.

Programmatic SEO Vs Answer Engine Optimization For SaaS

Nov 1, 2026 · 11 min read

Programmatic SEO Vs Answer Engine Optimization For SaaS

Programmatic SEO builds templated pages at scale. Answer engine optimization builds pages AI engines cite. We tested both on ChatGPT, Perplexity and Google's AI Overviews to see which pages actually get cited, and when SaaS teams should use each.

AI Search Optimization Vs ASO: Why The Acronym Is Confusing

Oct 30, 2026 · 10 min read

AI Search Optimization Vs ASO: Why The Acronym Is Confusing

AI search optimization vs ASO: why ASO now has four meanings, what Google and Microsoft call the work, how app stores changed, and a simple test for naming it.

Generative Engine Optimization GitHub Resources In 2026: What Is Useful And What Is Not

Oct 30, 2026 · 11 min read

Generative Engine Optimization GitHub Resources In 2026: What Is Useful And What Is Not

The generative engine optimization GitHub repos worth your time in 2026: the research code, the benchmark that tested GEO tricks, and repos to use with care.

ChatGPT Prompts For SEO Keyword Research That Do Not Create Generic Content

Oct 30, 2026 · 10 min read

ChatGPT Prompts For SEO Keyword Research That Do Not Create Generic Content

We tested four ChatGPT prompts for SEO keyword research against real search data. Which prompts found usable keywords, and which ones made up the numbers.

What Is GEO In Digital Marketing

Oct 30, 2026 · 8 min read

What Is GEO In Digital Marketing

What is GEO in digital marketing? It now usually means generative engine optimization, not geo-targeting: what it covers, where it came from, how to start.

THE SIGNAL

Insights on AI and Google search strategy. No fluff.

Get the latest on AI citations, Google rankings, and content strategy.

No spam. Unsubscribe anytime.