Someone asked r/SEO in September whether anyone had re-tested the claim that AI crawlers don't run JavaScript. Every source they followed led back to one Vercel analysis from December 2024. One reply said the answer depends on how much JavaScript a page has. Another pointed out that a person had just watched Claude summarize a JavaScript page using only its metadata.
Has anyone re-tested whether AI crawlers render JavaScript since Dec 2024?
Two things I keep reading everywhere and cannot verify. I would rather ask than repeat them. "AI crawlers don't execute JavaScript." Every citation I follow ends at the same Vercel analysis from 17 December 2024. That is 20 months old, and...
That's the problem with technical SEO in 2026. The checklists were written for Googlebot, and AI crawlers behave differently. This one covers both, and we ran it on 50 real business sites to see where they trip.
What we checked on 50 sites
On October 19, 2026, we audited the homepages of 50 business sites, made up of 30 software companies, 13 online stores and 7 local service businesses. We fetched each homepage, its robots.txt, sitemap and llms.txt, and requested the homepage again while labeled as OpenAI's, Perplexity's and Anthropic's crawlers.
| Check | Result |
|---|---|
| Homepage loaded for a normal request | 41 of 50 |
| robots.txt blocking AI search crawlers | 1 of 44 we could read |
| robots.txt naming AI crawlers at all | 10 of 44 |
| Homepage text missing without JavaScript | 1 of 41 |
| Sitemap found | 40 of 41 |
| Structured data (JSON-LD) on the homepage | 33 of 41 |
| Exactly one H1 heading | 32 of 41 |
| llms.txt file | 28 of 41 |
Most sites passed the basics. The trouble sat in places a status code or a robots.txt check won't show you.
AI search traffic grew 835% this year. Is your content ready?
RankControl generates 26 content formats optimized for ChatGPT, Claude, and Perplexity. Published on your domain, matched to your brand.
1. Crawler access in robots.txt
Start with robots.txt, since it's the one place every crawler checks first. Look for rules that block Googlebot, Bingbot or the AI search crawlers, which are OpenAI's OAI-SearchBot, Perplexity's PerplexityBot and Anthropic's Claude-SearchBot.
In our audit, only one site blocked the AI search crawlers. Its robots.txt blocked every crawler we checked, Googlebot included. Ten sites named AI crawlers in their file. Apart from that one site, none of them blocked the AI search crawlers from the homepage.
Know which crawler does what before you block anything. OpenAI says blocking GPTBot stops training but doesn't remove you from ChatGPT search, while blocking OAI-SearchBot does. Our complete AI crawler list covers each one.
2. Crawler access through your firewall
This is where we found the real problems. Nine of the 50 sites refused our plain scripted request with a 403 error, and six of those also refused the requests labeled as AI crawlers. Online stores were hit hardest, with 5 of 13 blocking us, against 1 of 30 software sites.
That doesn't prove the real crawlers are blocked. OpenAI, Perplexity and Anthropic publish the IP ranges their crawlers use, and firewalls often let verified crawlers through while blocking scripts like ours. Brooks Running refused all our requests, yet ChatGPT linked to Brooks product pages in a shopping test we ran the same day.
The only reliable test is your own data. Check your server logs for visits from the AI crawlers, and check your CDN's bot settings. Cloudflare changed its defaults for AI and mixed-use crawlers in September 2026, so a setting you never touched may have changed. Our guide to auditing AI crawlers in server logs walks through it.
3. The soft block
We'd call this one a soft block, a page that returns a normal 200 status but serves an error page instead of your content. Status-code checks miss it completely.
L.L.Bean's homepage gave our normal request about 3,100 words of text. The same request labeled as OpenAI's search crawler got a 200 status and 77 words, which turned out to be a firewall error page with an incident number. Any tool that only checked the status would have marked it as fine.
To catch soft blocks, compare what a crawler request gets with what a browser gets, and count the words as well as checking the status. A sudden drop in page size for one crawler is the clue.
Built by the team that got cited in 48 hours.
Content generation, backlink building, AI visibility tracking, and Google rankings. One platform, zero guesswork.
4. Content that loads without JavaScript
Google renders JavaScript, so a page built in the browser can still rank. Most AI crawlers don't. Vercel's analysis of crawler traffic found that ChatGPT's and Claude's crawlers fetch JavaScript files but don't run them, while Gemini uses Google's rendering.
As the Reddit thread points out, that study is almost two years old, and the vendor sells server-side rendering. But we couldn't find newer public evidence that the big AI crawlers run JavaScript, so build as if they don't.
In our audit, one of the 41 homepages had no readable text in its raw HTML, only a script that built the page in the browser. The median homepage carried about 1,550 words in plain HTML. To check your own pages, view the page source or fetch it without a browser and see whether your main text is there.
5. Indexing signals
These checks matter to Google and AI crawlers alike, since both rely on the same signals to decide what a page is.
- Status codes. Key pages should return 200, and moved pages should 301 to the new address.
- Noindex tags. None of the 41 homepages had one, but check templates after every release.
- Canonical tags. 2 of 41 homepages had none.
- Redirect chains. Keep them to one hop.
6. Sitemaps
A sitemap helps every crawler find your pages. 40 of the 41 sites we could load had one at /sitemap.xml or listed in robots.txt. Submit yours to Google Search Console and Bing Webmaster Tools, since Bing's index feeds Copilot and is one of ChatGPT's sources.
7. Titles, headings and structured data
Every site we loaded had a title and a meta description. Headings were messier. 32 of 41 had exactly one H1, 7 had more than one and 2 had none. Google's John Mueller has said multiple H1s are fine, but one clear H1 makes the page's topic obvious to any reader, human or crawler.
Structured data was common, on 33 of 41 homepages. Use JSON-LD for your organization, products, articles and FAQs, and keep it matching what's visible on the page.
8. Speed
Our full homepage fetch took a median of 0.84 seconds from our test machine, and 7 of 41 took over 1.5 seconds. Google's Core Web Vitals measure what users experience, which our check doesn't. Our post on Core Web Vitals and rankings covers what Google actually measures.
Speed matters for AI fetchers too, because some fetch pages live while answering a question. A slow server makes it harder for those live fetches to get your content in time.
9. llms.txt
28 of the 41 sites had an llms.txt file, mostly software companies. No major AI engine has confirmed it uses the file to choose sources, so treat it as optional. Our robots.txt and llms.txt checklist covers how to set both up.
The checklist on one page
Here's the whole audit in one table, with the quickest way to test each item.
| Check | Why Google cares | Why AI crawlers care | Quick test |
|---|---|---|---|
| robots.txt rules | Blocks keep pages out of Search | Blocks keep pages out of AI answers | Read the file for each crawler name |
| Firewall and CDN | Rarely blocks Googlebot | Often blocks scripted fetches | Server logs and CDN bot settings |
| Soft blocks | Error pages get indexed as content | Error pages replace your content | Compare word counts by user agent |
| JavaScript | Google renders it | Most AI crawlers don't | View source for your main text |
| Indexing signals | Decide what gets indexed | Decide which version gets read | Status, canonical and noindex checks |
| Sitemap | Helps discovery | Helps discovery | Open /sitemap.xml |
| Headings and schema | Clarify the page topic | Make facts easy to lift | Check H1 and JSON-LD |
| Speed | Part of page experience | Live fetches need a fast reply | Time a full page fetch |
How to prioritize what you find
Audits can produce hundreds of warnings. A thread in r/TechSEO in August asked how to rank them, and the best replies said to group duplicates into single fixes and weigh each by how many pages and how much traffic it touches.
Hundreds of findings found in the audit. How to prioritize the technical SEO issues?
Large technical SEO audits often produce hundreds or even thousands of findings but treating every issue as equally important doesn't look practical. For example, I might have crawlability issues, indexation inconsistencies, duplicate URLs,...
We'd order this checklist the same way. Access problems come first, because a blocked crawler sees nothing else on your site. JavaScript and indexing come next, and headings, schema and llms.txt last.
Limits of our audit
We checked homepages only, once, from one server. Our requests copied crawler user agents but came from our own IP address, so firewalls could treat them differently from the real crawlers. The 50 sites came from our earlier studies, so they lean toward software companies and online stores.
What this takes by hand
Here's a rough budget for a technical audit of a small site, by our estimate.
| Task | Hours |
|---|---|
| Robots.txt, firewall and log checks | 2 to 3 |
| JavaScript and soft-block checks on key templates | 2 to 3 |
| Indexing, sitemap and heading checks | 2 to 4 |
| Structured data and speed checks | 2 to 3 |
| Writing up and prioritizing fixes | 2 to 3 |
| Total for one audit | 10 to 16 |
Doing it by hand or with help
You can run this audit yourself once or twice a year. The part that changes without warning is crawler access, since firewall and CDN defaults move on their own. RankControl tests every day whether the AI crawlers can reach your site and flags a block when it finds one. Its AI visibility tracking checks 50 of your buyer questions every week on ChatGPT, Perplexity, Claude, Gemini, Grok, Google AI Mode and Google AI Overviews. If you run a software site, our technical SEO guide for SaaS goes deeper on templates and docs.
See your first AI citation report in under 5 minutes.
No setup calls. No onboarding meetings. Connect your domain and see where AI mentions your brand right now.
