JavaScript rendering and generative engines: what AI crawlers actually see.

An AI crawler reads the initial HTML, not the rendered page

AI search crawlers read the HTML document your server returns and stop there. To our knowledge, neither OpenAI, Anthropic nor Perplexity documents JavaScript execution by their agents, and the available observations point the same way: an analysis published by Vercel in late 2024, based on its network logs, concluded that the crawlers of these three providers sometimes downloaded script files without executing them. Our own page-by-page tests do not contradict this finding. It may change; until proven otherwise, the working rule is simple: what is not in the initial HTML does not exist for a generative engine.

Google is the exception. Googlebot has a rendering service based on a recent version of Chromium, which executes JavaScript in a second pass, after crawling, with a variable delay. Bingbot renders pages too. This explains the typical case we come across: a single-page application well represented in Google's index, and in the AI Overviews that draw on it, but absent from ChatGPT, Claude and Perplexity.

The consequence goes beyond single-page applications. Many conventional sites, built on a CMS that returns complete HTML, then inject in JavaScript precisely the elements that make a source valuable: prices, reviews, the answers in an accordion FAQ, structured data. The page looks complete and the citable part is missing.

The ten-minute test

The test consists of fetching the page the way an AI search crawler does, then checking that the content to be cited is there. It needs neither a paid tool nor server access, only a command line. The first command saves the initial HTML while identifying as OAI-SearchBot; the second searches for a sentence you know is visible on screen; the third gives an order of magnitude for the text received, to compare with the visible text.

# 1. The initial HTML, as an AI search crawler receives it
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  https://www.exemple.fr/guide/delais-de-livraison/ -o initial.html

# 2. Is a sentence visible on screen in there? (0 = missing)
grep -c "livraison sous 48 heures" initial.html

# 3. Order of magnitude of the text received, tags excluded
sed 's/<[^>]*>//g' initial.html | tr -s ' \n' | wc -w

# 4. Is the structured data in the document?
grep -c 'application/ld+json' initial.html

The same check can be done without a command line: in the browser, “view page source” shows the initial HTML, while the inspector shows the document after execution. Disabling JavaScript in the developer tools and reloading the page gives a view close to an AI crawler's. By contrast, the URL inspection tool in Search Console shows the page as rendered by Googlebot; it is falsely reassuring about the other engines.

What to check, in order: the main text, section headings, FAQ answers, prices and availability, tables and lists, structured data. For each element the answer is binary: present in initial.html, or absent. On a sample of twenty representative pages, the result usually points to one or two faulty templates, rarely the whole site.

The eight patterns that empty a page

The causes of absence repeat from one site to the next. Not all of them stem from a framework choice: several affect sites built on conventional CMSs, through an accumulation of modules and third-party scripts.

Table: scroll horizontally.

The case of structured data deserves emphasis. Many teams have got into the habit of adding JSON-LD through a tag manager, because it is quick and Google reads it after rendering. For a crawler that executes nothing, this block does not exist: the entity, the author and the marked-up FAQ vanish in one go. The remedy is simple, and it is usually the first fix we ask for: write the JSON-LD into the template, server-side.

Why Google copes, and why that does not protect you

Google renders pages because it has the resources and the head start: its rendering service executes JavaScript at web scale, in a queue separate from crawling. Rendering is not free, though. It happens with a delay, it fails when a resource is blocked or too slow, and it consumes a budget that Google allocates according to the importance it attaches to the site. A client-rendered page is therefore indexed later, and sometimes incompletely, even at Google.

For the other engines, the question does not arise: with no rendering, the page is read as it is. And the four conditions described in our method for getting cited are sequential; an inaccessible page is never assessed on its quality. A site can thus meet the first condition for Google and fail it for all the others, with the same code and the same robots.txt. This is one of the cases where citation share, measured engine by engine, reveals a problem that conventional SEO does not show: a good presence in AI Overviews, absence elsewhere.

A nuance on user-triggered fetchers. ChatGPT-User, Claude-User and Perplexity-User visit a page when a user asks for it; nothing indicates that they render JavaScript any more than the index crawlers do. The link you paste into a conversation to “make the model read” your page is subject to the same limit.

The solutions, from the lightest to the most structural

The principle common to every solution is the same: deliver the citable content in the initial HTML, and keep JavaScript for interactivity. How to get there depends on the technical stack and on what can be changed.

The common frameworks offer these modes natively: Next.js, Nuxt, SvelteKit and Angular provide server rendering and static generation, and Astro is built around islands. Migration does not require changing framework; it requires deciding, template by template, which rendering mode applies, and moving to the server the data calls that feed the citable content.

That leaves dynamic rendering: serving crawlers a version pre-rendered by a headless browser, and serving users the application as it is. Google describes it as a workaround, acceptable as long as the content served is identical. It has two weaknesses specific to generative engines. The list of agents to recognise must be kept up to date with every new agent, and the rendering lags behind the pages. It remains a reasonable choice to buy time on an application that cannot be rebuilt in the short term.

Prioritising: which pages first

Not the whole site needs to be readable by a generative engine. A business application, a customer area or a configurator will never be cited, and do not need to be. The pages to deal with are those that answer your buyers' questions: guides, comparisons, FAQs, product pages with their specifications, pricing pages, local pages. The prompt panel used to measure citation share provides the list: each question designates the page that should answer it, and that page takes the test.

Measurement validates each phase. After the fixes, the logs must show the AI search crawlers returning to the corrected pages with 200 responses of normal size; a few weeks later, citation share on the corresponding questions must move, engine by engine. If the logs show no visits at all, the problem is not rendering but access, and you need to look at robots.txt or a firewall rule, which our article on accidental blocking covers.

The traps after migration

A move to server rendering guarantees nothing until acceptance testing has been done with the initial HTML test. The following errors recur in our audits of recently migrated sites.

• A server-rendered template whose main component still loads its text through an API call on mount: the shell is complete, the body arrives afterwards.
• Loading “skeleton” blocks served in the initial HTML in place of the content, then replaced at execution.
• Hydration that replaces the rendered content with a different version, and initial HTML that no longer matches what the user sees.
• A CDN cache that keeps serving the old empty shell to requests without browser headers.
• Dynamic rendering whose agent list ignores OAI-SearchBot, Claude-SearchBot or PerplexityBot, which are newer than the list.
• A JavaScript challenge imposed by the firewall on non-human requests: a crawler that executes nothing will never pass it, and receives a challenge page instead of the content.
• A server response time degraded by rendering, which pushes crawlers to give up on slow pages.

What to remember