Can AI crawlers see JavaScript content?

Most AI crawlers don't run JavaScript. What that means for React and Next.js sites, and how to make sure your content is in the HTML.

Updated 2 min readBy Satchit
Short answer

Mostly no. A 2024 study of crawler traffic by Vercel and MERJ found that OpenAI's crawlers, ClaudeBot, PerplexityBot, Meta's crawler, Bytespider and CCBot download JavaScript files but don't execute them. Google's crawlers, which Gemini also uses, and Applebot do render JavaScript.

So if your page builds its content in the browser, ChatGPT, Claude and Perplexity see an almost empty page.

What's at risk#

Content that appears only after JavaScript runs:

  • Text loaded with fetch inside useEffect
  • Prices, plans or product details pulled from an API on the client
  • Tabs, accordions and "read more" sections that are only added to the page when clicked
  • Whole apps built as a client-rendered single page with an empty <div id="root">

Content that is in the HTML, even if it's hidden visually (a closed accordion or <details> element), is fine.

Test your own siteSee which AI crawlers your site lets in and what else stops them quoting you.

Fixing it in Next.js#

Render on the server. In the App Router, components are server components by default, so data you fetch in them ends up in the HTML:

// app/pricing/page.tsx: rendered on the server
export default async function Pricing() {
  const plans = await getPlans()
  return <PlanTable plans={plans} />
}

Keep client components for interaction, and pass them data that was fetched on the server. For pages that rarely change, static generation is best: the HTML is built once and served instantly.

Check a page#

Open view-source on the page and search for a sentence from the body. Or run the checker: its "readable without JavaScript" check counts the words in the raw HTML and flags pages where most of the content is missing.

Source: Vercel, The rise of the AI crawler.

Questions

Does Googlebot run JavaScript?

Yes. Googlebot renders pages, and Gemini uses the same infrastructure. Most other AI crawlers don't.

Is a Next.js site automatically fine?

Only for content rendered on the server. Anything fetched in useEffect or behind a client component that loads data after the page opens is missing from the HTML.

How do I see what a crawler sees?

View the page source (not the inspector), or run curl on the URL. If the text isn't there, most AI crawlers can't read it.

Related guides