What Is An LLM Search Engine And How Does It Pick Sources?

What an LLM search engine is, the four steps behind every answer, and a dated table of which crawler and index runs under ChatGPT, Perplexity, Claude, Gemini, Grok and AI Mode.

RankControl10 min read
What Is An LLM Search Engine And How Does It Pick Sources?

Someone on r/NoStupidQuestions asked what the real difference is between googling something and asking ChatGPT. The top-voted reply put it in one line. Google is an index of links to answers, and ChatGPT can use the same kind of index but does one more step, which is to read what it found and write you an answer.

r/NoStupidQuestions· u/angelkiye· Oct 14, 2026

Fundamentally what’s the difference between Googling something and asking ChatGPT?

Couple of disclaimers first:- I am very much anti-AI- I’ve never used ChatGPT and don’t plan on it either I’m not concerned with the ethics of AI usage with this particular question only. Also this question is more in reference to simple, s...

↑ 0 upvotes30 comments
Via Reddit

That one more step is what makes an LLM search engine, and it hides the part most people never ask about. Whose index is it reading? This guide defines the term, then lays out what runs underneath each of the six big engines, with what each vendor documents and what's only inferred. I read every vendor page cited here on October 19, 2026.

What an LLM search engine is

An LLM search engine is a large language model joined to a search system. It decides whether a question needs fresh facts, turns the question into one or more searches, filters what comes back, and writes a single answer that cites the pages it used. ChatGPT search, Perplexity, Claude with web search, Gemini, Grok and Google's AI Mode all work this way.

You'll see the same four steps in each vendor's docs, under different names. For the model underneath, see how large language models work.

StepWhat happensWhere a vendor documents it
DecideThe model judges whether to search at allAnthropic says Claude "determines when to search based on the prompt"
Rewrite and fan outThe question becomes several searchesGoogle's AI Mode issues "multiple related searches concurrently across subtopics"
Retrieve and rerankResults come back and get filteredWith Anthropic's newer web search tool, Claude writes code to filter results before they reach its context
Ground and citeThe answer is written from the results, with linksGemini's API returns inline citations tied to the parts of the answer they support

According to Microsoft's Copilot Studio docs, Copilot Studio agents run the same loop on public sites. It sends grounding queries to the Bing index and scores what comes back.

Each step is a place where your page can drop out. A question that skips search never reaches your page. A fan-out query you don't rank for won't either, and a page that survives retrieval can still be filtered out before the model writes a word.

How it differs from a classic search engine

A classic search engine ranks pages and stops. You get ten links and do the reading yourself. An LLM search engine does the reading for you, so the list you used to judge gets replaced by one answer with a few sources attached.

The thing most explainers skip is that the model isn't the part doing the searching. I find it useful to split every LLM search engine into two layers. The answer layer is the model that writes and cites. The index layer is the crawler and search index that supplies the pages. The answer layer gets the headlines, but the index layer decides which pages the model ever sees, and it runs on crawlers you can name in robots.txt.

Some engines own both layers. Others rent the index layer from a partner, and a few won't say. If you're new to the jargon around this, our guide to LLM SEO, GEO, AEO and AIO sorts out the names.

RANKCONTROL

Know exactly what AI says about your competitors.

RankControl's Recon Agent monitors competitor citations across ChatGPT, Perplexity, Claude, Gemini, Grok, Google AI Mode, and Google AI Overviews. See where they show up and you don't.

What runs underneath each LLM search engine

This is the index layer for the six engines, checked against each vendor's own pages on October 19, 2026. "Documented" means the vendor says it in its own docs. "Inferred" means the evidence is strong but the vendor's product docs don't state it.

EngineCrawler or fetcherIndex or search partnerStatusSource and date
ChatGPTOAI-SearchBot fills OpenAI's own index; ChatGPT-User fetches pages for user actionsOpenAI's index plus outside partners; Bing named for Enterprise and EduCrawler documented; Bing documented for Enterprise and Edu only; the mix per answer not publishedOpenAI crawler docs and Help Center, read Oct 19, 2026
PerplexityPerplexityBot surfaces and links sites; Perplexity-User visits pages when a user asksIts own index; no partner namedCrawler documented; any licensed index reported only by third partiesPerplexity crawler docs, read Oct 19, 2026
ClaudeClaude-SearchBot improves search quality; Claude-User visits pages for usersBrave Search, listed as a web search subprocessorInferred; the web search tool docs name no providerAnthropic crawler page (updated Apr 7, 2026), web search docs, Trust Center
GeminiGooglebot, through the Google Search indexGoogle Search, used to ground Gemini answersDocumentedGoogle crawler page (updated Oct 15, 2026), Gemini API docs (updated Oct 13, 2026)
Google AI ModeGooglebot, through the Google Search indexGoogle Search, with query fan-outDocumentedGoogle Search blog, Mar 5, 2025
GrokNo crawler or robots.txt page foundWeb Search and X Search tools; index not namedTools documented; index undocumentedxAI tool docs, read Oct 19, 2026

A few rows need more than a cell.

ChatGPT

OpenAI's crawler docs say OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features" and that robots.txt changes take about 24 hours to apply. The same page never mentions Bing. That comes from OpenAI's Help Center, which says Enterprise and Edu workspaces may share search queries with Bing. How the mix shifted between Bing, Google results and OpenAI's own index over 2025 and 2026 is a long story, and does ChatGPT search use Bing tells it in full.

Perplexity

Perplexity's crawler page describes PerplexityBot as built "to surface and link websites in search results on Perplexity" and says it isn't used to train models. It says nothing about Bing or any other partner. Some outside reporting claims Perplexity tops up its index with licensed results, but I couldn't find Perplexity saying that, so the table marks it unconfirmed.

Claude

This is the row to mark carefully. Anthropic's web search tool docs cover versions, pricing at $10 per 1,000 searches, and citations of up to 150 characters, but name no search provider. Its crawler help page lists Claude-SearchBot and Claude-User and doesn't name one either. The evidence sits in Anthropic's Trust Center instead.

Anthropic Trust Center subprocessor list filtered to Brave Search, showing Brave Search listed for Web Search, covering all products, located in the United States

A subprocessor list shows who handles data. It doesn't show how results get chosen, and Brave could be one source among several, which is why the table says inferred.

Gemini and AI Mode

Google owns both layers, but they don't share one switch. Google's crawler page says the Google-Extended token controls whether your content is used for "grounding" in Gemini Apps. The same page says Google-Extended "does not impact a site's inclusion in Google Search," and AI Mode is part of Search. So blocking Google-Extended can take you out of Gemini's grounded answers while AI Mode keeps citing you. On the API side, Gemini's grounding docs say the model "automatically generates one or multiple search queries and executes them" when it judges a search would help.

Grok

xAI's web search docs say the tool can "search the web in real-time and browse web pages," and Grok also has an X Search tool for posts on X. Neither page names an index. Grok was the only engine of the six where I couldn't find a published crawler name or robots.txt rule, so there's no documented way to opt out.

How an LLM search engine picks sources, briefly

Once the index layer hands over candidates, the answer layer chooses. The details differ by engine, and other guides here go deeper, so this part stays short.

The best academic test of what helps is the GEO paper from Princeton and partners, presented at KDD 2024. Adding citations, quotes and statistics to a page lifted its visibility in generated answers by up to 40%. One caveat the headline skips is that the team tested a generative engine they built in the style of Bing Chat, then checked results on Perplexity. It points the right way, but it measured a lab engine, so today's production engines may weigh things differently.

Freshness seems to count too. Ahrefs, which sells SEO tools, looked at about 17 million cited URLs and found that pages cited by AI assistants averaged 1,064 days old, against 1,432 days for plain organic results. Google's engine leans hardest on its own rankings, and that link is getting tighter. BrightEdge, another SEO vendor, tracked AI Overview citations that also rank organically and saw the share climb from 32.3% to 54.5% over 16 months.

For depth, go engine by engine:

The answer layer also changes how sources look, and Google changed it again in October. On October 15, Google said AI Overviews and AI Mode would show links higher up in responses and more often, as The Verge reported that day.

Links will appear ‘higher up in responses more frequently’ in Google’s AI Mode and AI Overviews. https://t.co/D06fi4pBEL

The Verge@vergeOct 15, 2026

Google's Robby Stein added that use of Preferred Sources, which lets people pick sites they want to see more of, has grown more than seven times since the start of the year.

RANKCONTROL

26 content formats. Published on your domain. Matched to your brand.

Guides, comparisons, listicles, case studies, and more. RankControl generates content that gets cited by ChatGPT, Perplexity, Claude, Gemini, Grok, Google AI Mode, and Google AI Overviews.

What the index layer means for your site

The table turns into a short to-do list. Each engine can only cite what its index layer can reach, so access comes first.

  1. Check robots.txt and your firewall for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot. Blocking any of them takes you out of that engine's index layer.
  2. Decide on Google-Extended on purpose. Blocking it limits Gemini's grounded answers but leaves Search, and so AI Mode, alone.
  3. Since Bing is a named ChatGPT partner, verify your site in Bing Webmaster Tools.
  4. If the Brave link holds, a cheap test for Claude is to search your key pages in Brave Search and see if they show up.
  5. For Grok, the only lever right now is being findable on the open web and active on X.

Our robots.txt checklist for AI crawlers has the exact user agents and rules.

The harder part is that this table won't hold still. Google updated its crawler page on October 15 and its Gemini grounding page two days before that. Claude's web search tool now comes in three versions, and the newest is dated March 2026. Fixing access once is easy. The real problem is knowing when an engine that used to cite you quietly stopped.

Keep track of all six engines

By hand, checking 50 buyer questions on seven engines every week takes about six hours. Add the time to notice that one engine dropped you, and work out whether it was the index layer or the answer layer. These are rough estimates.

RankControl's AI visibility tracking runs your 50 questions every week on ChatGPT, Perplexity, Claude, Gemini, Grok, Google AI Mode and Google AI Overviews. Our tracking marks each answer as mentioned, cited or both, and records which competitors and sources each engine used.

You can keep this table yourself with a spreadsheet, a robots.txt check and a weekly round of questions. Or RankControl's agents can run the weekly checks across seven engines, those six among them, while you work on the pages they should be citing.

Your competitors are getting cited by AI. You're not.

Every day without citation tracking is a day your competitors pull ahead in ChatGPT, Perplexity, and Claude.

Show me who's getting cited →2-minute overview · real case-study numbers

Frequently Asked Questions

A classic search engine ranks existing pages and hands you a list of links. An LLM search engine runs searches for you, reads and filters the results, then writes one answer and cites the pages it used. Many of them still sit on top of a classic index.

OpenAI's own index, filled by its OAI-SearchBot crawler, plus outside partners. OpenAI's help page for Enterprise and Edu plans names Bing as one of them. OpenAI doesn't publish how much each source adds to a given answer.

Probably, but Anthropic hasn't said so in its web search docs. Brave Search appears on Anthropic's subprocessor list under web search for all products, which is why it's widely treated as Claude's backend. Treat it as inferred, not confirmed.

Both draw on Google's Search index, but the opt-outs differ. Blocking Google-Extended in robots.txt stops your content being used to ground answers in Gemini Apps, while Google says it has no effect on Google Search, which includes AI Mode.

There's no documented way yet. xAI's docs describe Grok's web search and X search tools, but as of October 19, 2026 there was no xAI page naming a crawler or a robots.txt rule to opt out.

RANKCONTROL

Turn AI search into a customer acquisition channel

Content that ranks on Google and gets cited by AI search engines. Published on your domain. Citations tracked weekly.

Related Articles

Answer Engine Optimization: The Complete 2026 Guide

Oct 30, 2026 · 10 min read

Answer Engine Optimization: The Complete 2026 Guide

Answer engine optimization in 2026: how ChatGPT, Perplexity, Claude and Google pick sources, the work that earns citations, what to skip, how to measure AEO.

What Is Perplexity In SEO? How The Answer Engine Changes Search Visibility

Sep 22, 2026 · 8 min read

What Is Perplexity In SEO? How The Answer Engine Changes Search Visibility

Perplexity is the citation-first answer engine. What it is, how its retrieval works, what it rewards, and how to earn a place in its answers.

How to Get Cited by AI Search Engines: The Master Guide

Nov 15, 2026 · 15 min read

How to Get Cited by AI Search Engines: The Master Guide

How to get cited by AI search engines, built on a dozen dated checks and each engine's own documentation: access, when answers search, what each engine cites, and how to track it.

How Does Generative Engine Optimization Work? The Simple Version For Marketers

Nov 12, 2026 · 9 min read

How Does Generative Engine Optimization Work? The Simple Version For Marketers

How generative engine optimization works, shown by tracing one buyer question through an AI answer: the search decision, the rewritten searches, the pages read, and the citations.

Generative Engine Optimization: The Definitive Guide

Oct 30, 2026 · 10 min read

Generative Engine Optimization: The Definitive Guide

A guide to generative engine optimization: where GEO came from, how AI engines pick sources, what works, what to skip and how to measure it, with 2026 data.

Perplexity Brain: How Agentic Memory Could Change Source Selection

Oct 30, 2026 · 6 min read

Perplexity Brain: How Agentic Memory Could Change Source Selection

Perplexity Brain gives agents a self-improving memory wiki. Why answer incumbency, refresh cycles, and evidence links change how sources get picked.

Self-Promotional Listicles And AI Citations: Do They Still Work?

Nov 10, 2026 · 15 min read

Self-Promotional Listicles And AI Citations: Do They Still Work?

Do 'best X' lists where a brand ranks itself first still get cited by ChatGPT, Perplexity and Google's AI answers? Five dated 2026 studies disagree. Here's what each measured, why they disagree, how engines differ, what Google says and what we saw ourselves.

AI Search Adoption Statistics 2026: Users, Queries, and Growth

Nov 8, 2026 · 10 min read

AI Search Adoption Statistics 2026: Users, Queries, and Growth

AI search adoption statistics for 2026: ChatGPT, Google AI Overviews, AI Mode, Gemini, Claude and Perplexity users, query volumes and who is growing fastest.

How To Monitor AI Citations And Brand Mentions Across AI Search

Nov 3, 2026 · 9 min read

How To Monitor AI Citations And Brand Mentions Across AI Search

How to monitor AI citations and brand mentions with the scheduled tasks ChatGPT, Perplexity, Google, Gemini, Claude and Grok now ship, checked against each one's docs, and where they fall short.

THE SIGNAL

Insights on AI and Google search strategy. No fluff.

Get the latest on AI citations, Google rankings, and content strategy.

No spam. Unsubscribe anytime.