Server-Side Rendering for AI Crawler Compatibility

Q: How do I test what AI crawlers see on my site?

Use curl to fetch your pages without JavaScript execution. Run 'curl -s https://yourdomain.com | head -50' and check if the response contains your actual content or just an empty div element with script tags. If you see content in the HTML, AI crawlers can see it. If you see an empty shell, your site needs server-side rendering.

There is a fundamental disconnect between how modern web applications work and how AI crawlers consume content. Modern frameworks like React, Vue, and Angular default to client-side rendering: the server sends a minimal HTML shell, JavaScript downloads, executes, fetches data, and paints the page. Users with modern browsers see the full page in under a second. AI crawlers see nothing.

This is not a minor issue. GPTBot, ClaudeBot, PerplexityBot, and most other AI crawlers do not execute JavaScript. They make a simple HTTP request, receive the HTML response, and process whatever text content is in that response. If your content is rendered by JavaScript after the initial page load, it does not exist as far as these crawlers are concerned.

The consequence is stark: a beautifully built React SPA with excellent content can be completely invisible to ChatGPT, Claude, Perplexity, and Google AI Overviews. Your competitors with server-rendered content appear in AI responses. You do not. This guide explains why this happens, how to diagnose it, and how to fix it across every major framework. Use our AgentReady scanner to check whether your site serves content in the initial HTML response.

Why AI crawlers cannot execute JavaScript

Understanding why AI crawlers skip JavaScript requires understanding the economics of web crawling. Google crawls and indexes hundreds of billions of pages. Running a headless browser with full JavaScript execution for every page would require extraordinary computational resources. Google invests in this because search indexing is their core business. They run a rendering service based on headless Chrome that executes JavaScript for pages in their crawl queue.

AI companies are in a different position. Their crawlers are designed to gather text content for training data and real-time search features. Running a headless browser for every page would multiply their infrastructure costs by 10-100x and dramatically slow their crawl rate. The pragmatic solution is simple: fetch the HTML, extract the text, move on.

This means AI crawlers operate like the simplest possible HTTP client. They send a GET request and read the response body. No DOM construction, no CSS evaluation, no JavaScript execution, no network requests triggered by scripts. What is in the HTML is what they get. Everything else is invisible.

What AI crawlers actually see

Here is what a typical React SPA serves as its initial HTML response:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>My App</title>
  <link rel="stylesheet" href="/static/css/main.abc123.css">
</head>
<body>
  <div id="root"></div>
  <script src="/static/js/main.def456.js"></script>
</body>
</html>

An AI crawler receives this and processes it. The entire content of the page, as far as the crawler is concerned, is: a title tag that says "My App" and an empty div. There is no description of your product, no pricing information, no case studies, no blog posts. Zero useful content.

Compare that to a server-rendered page:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>AI-Powered Product Studio | p0stman</title>
  <meta name="description" content="We build AI agents, websites, and mobile apps...">
</head>
<body>
  <header>...navigation with links...</header>
  <main>
    <h1>AI-Powered Product Studio</h1>
    <p>We build custom AI agents, web applications, and mobile apps
    for growing businesses. Based in London, UK.</p>
    <section>
      <h2>Our Services</h2>
      <p>AI Agent Development from GBP 5,000...</p>
      ...hundreds of lines of real content...
    </section>
  </main>
  <script src="/static/js/main.def456.js"></script>
</body>
</html>

Same page, same visual result for users. But the AI crawler now has access to every piece of content, every heading, every price, every link. This is the difference between being visible and being invisible in AI search.

Rendering strategies compared

There are four main rendering strategies, each with different implications for AI crawler visibility:

Strategy	When HTML is generated	AI crawler visibility	Best for
CSR (Client-Side Rendering)	In the browser, after JS loads	None -- empty shell	Authenticated dashboards only
SSR (Server-Side Rendering)	On the server, per request	Full content visible	Dynamic, personalised pages
SSG (Static Site Generation)	At build time	Full content visible	Content that rarely changes
ISR (Incremental Static Regeneration)	At build time + background revalidation	Full content visible	Content that changes periodically

CSR: Client-Side Rendering

CSR is the default for vanilla React (create-react-app), Vue CLI, and Angular CLI. The server sends a minimal HTML file with script tags. JavaScript downloads, executes, makes API calls, and renders the page entirely in the browser.

AI crawler impact: Catastrophic. Zero content is available in the initial HTML. Your entire site is invisible to GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers. The only text they can read is your title tag and meta description (if you have static ones in the HTML template).

When CSR is acceptable: Only for authenticated application dashboards that should not be indexed by any crawler. Login pages, admin panels, internal tools.

SSR: Server-Side Rendering

SSR generates the complete HTML on the server for every request. The server executes your application code, fetches any required data, renders the component tree to HTML, and sends the full page to the client. JavaScript then "hydrates" the page to add interactivity.

AI crawler impact: Full content is visible in the initial HTML response. Every piece of text, every heading, every link is available to crawlers.

Trade-offs: Higher server load than SSG because HTML is generated per request. Slower Time to First Byte (TTFB) if data fetching is slow. Requires a server or serverless function to handle requests (cannot be deployed to a static CDN alone).

SSG: Static Site Generation

SSG generates all HTML pages at build time. The build process runs your application, fetches data, renders every page to static HTML files, and outputs them to a directory. These files are served directly by a CDN with no server processing per request.

AI crawler impact: Full content visible. Identical to SSR from the crawler's perspective -- the HTML contains all content.

Trade-offs: Content can only change by rebuilding and redeploying. Not suitable for highly dynamic content (user-generated content, real-time data). Build times increase with page count. Excellent performance because pages are served from CDN cache.

ISR: Incremental Static Regeneration

ISR combines SSG and SSR. Pages are generated at build time (like SSG) but can be revalidated in the background after a configured time interval. When a request comes in for a stale page, the cached version is served immediately while a new version is generated in the background for the next request.

AI crawler impact: Full content visible. The crawler always gets a pre-rendered HTML page.

Trade-offs: Content can be slightly stale (up to the revalidation interval). Requires a server or serverless functions for the background revalidation process. Excellent balance of performance and freshness for most websites.

How to detect the problem

The simplest way to see what AI crawlers see is to use curl, which makes a plain HTTP request without JavaScript execution -- exactly like an AI crawler.

The curl test

# Fetch your homepage and look at the HTML
curl -s https://yourdomain.com | head -100

# Search for actual content in the response
curl -s https://yourdomain.com | grep -i "your product name"

# Check for the empty SPA shell pattern
curl -s https://yourdomain.com | grep 'id="root"'
curl -s https://yourdomain.com | grep 'id="app"'
curl -s https://yourdomain.com | grep 'id="__next"'

If curl returns a page full of your actual content (headings, paragraphs, product descriptions), your site is server-rendered and AI crawlers can see it. If it returns an HTML shell with just a <div id="root"></div> and script tags, your content is invisible to AI crawlers.

The view-source test

In Chrome, right-click on your page and select "View Page Source" (not "Inspect Element"). View Page Source shows the raw HTML that was delivered by the server -- the same HTML that AI crawlers receive. Inspect Element shows the live DOM after JavaScript has executed, which includes all your content regardless of rendering strategy.

If View Page Source shows your content, you are server-rendered. If it shows an empty shell, you are client-rendered.

The AgentReady scanner

Our AgentReady scanner automates this check. It fetches your page without JavaScript execution and analyses the response for content. It flags sites that serve empty SPA shells and recommends specific fixes based on your framework.

Framework-specific SSR implementation

Next.js (React)

Next.js is the recommended React framework for AI-visible websites because it defaults to server rendering in the App Router.

App Router (recommended): All components are React Server Components by default. They render on the server and send HTML to the client. Only components with the "use client" directive render on the client.

// app/page.tsx - This is a Server Component by default
// Content is in the HTML response automatically

export default async function HomePage() {
  // Data fetching happens on the server
  const services = await getServices();

  return (
    <main>
      <h1>Our Services</h1>
      {services.map((service) => (
        <section key={service.id}>
          <h2>{service.name}</h2>
          <p>{service.description}</p>
          <p>From {service.price}</p>
        </section>
      ))}
    </main>
  );
}

Pages Router: Use getServerSideProps for SSR or getStaticProps for SSG:

// pages/index.tsx - Pages Router SSR
export async function getServerSideProps() {
  const services = await getServices();
  return { props: { services } };
}

export default function HomePage({ services }) {
  return (
    <main>
      <h1>Our Services</h1>
      {services.map((service) => (
        <section key={service.id}>
          <h2>{service.name}</h2>
          <p>{service.description}</p>
        </section>
      ))}
    </main>
  );
}

Static generation with ISR:

// app/blog/[slug]/page.tsx - ISR in App Router
export const revalidate = 3600; // Revalidate every hour

export default async function BlogPost({ params }) {
  const post = await getPost(params.slug);
  return (
    <article>
      <h1>{post.title}</h1>
      <div>{post.content}</div>
    </article>
  );
}

export async function generateStaticParams() {
  const posts = await getAllPosts();
  return posts.map((post) => ({ slug: post.slug }));
}

Nuxt (Vue)

Nuxt 3 defaults to universal rendering (SSR + client hydration). Content is server-rendered automatically.

<!-- pages/index.vue -->
<template>
  <main>
    <h1>Our Services</h1>
    <section v-for="service in services" :key="service.id">
      <h2>{{ service.name }}</h2>
      <p>{{ service.description }}</p>
    </section>
  </main>
</template>

<script setup>
// useFetch runs on the server during SSR
const { data: services } = await useFetch('/api/services');
</script>

For static generation, set the route rule in nuxt.config.ts:

// nuxt.config.ts
export default defineNuxtConfig({
  routeRules: {
    '/blog/**': { swr: 3600 }, // ISR with 1-hour revalidation
    '/about': { prerender: true }, // Static at build time
  }
});

SvelteKit (Svelte)

SvelteKit defaults to SSR. Server-side data loading uses the load function in +page.server.ts:

// src/routes/+page.server.ts
export async function load() {
  const services = await getServices();
  return { services };
}

// src/routes/+page.svelte
<script>
  export let data;
</script>

<main>
  <h1>Our Services</h1>
  {#each data.services as service}
    <section>
      <h2>{service.name}</h2>
      <p>{service.description}</p>
    </section>
  {/each}
</main>

For static pre-rendering, add the prerender option:

// src/routes/about/+page.ts
export const prerender = true;

Remix (React)

Remix is SSR by default. Data loading uses the loader function:

// app/routes/_index.tsx
import { json, type LoaderFunctionArgs } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

export async function loader({ request }: LoaderFunctionArgs) {
  const services = await getServices();
  return json({ services });
}

export default function Index() {
  const { services } = useLoaderData<typeof loader>();
  return (
    <main>
      <h1>Our Services</h1>
      {services.map((service) => (
        <section key={service.id}>
          <h2>{service.name}</h2>
          <p>{service.description}</p>
        </section>
      ))}
    </main>
  );
}

Astro

Astro renders pages to static HTML by default and supports multiple UI frameworks (React, Vue, Svelte). It is designed for content-heavy sites and produces zero JavaScript by default:

---
// src/pages/index.astro
const services = await getServices();
---

<html lang="en">
<body>
  <main>
    <h1>Our Services</h1>
    {services.map((service) => (
      <section>
        <h2>{service.name}</h2>
        <p>{service.description}</p>
      </section>
    ))}
  </main>
</body>
</html>

React Server Components and AI visibility

React Server Components (RSC), introduced with Next.js 13 App Router, represent a significant shift in how React applications render content. In the traditional React model, all components render on the client. With RSC, components render on the server by default.

This has a direct positive impact on AI crawler visibility:

Server Components render to HTML on the server. The full content is in the initial HTML response.
Data fetching happens on the server. No client-side API calls needed for initial content.
Zero JavaScript is sent for Server Components. The component logic stays on the server; only the HTML output is sent to the client.
Client Components are still hydrated. Interactive elements work for users after JavaScript loads, but the initial HTML already contains the content.

The rule is simple: if an AI crawler needs to see it, make it a Server Component. If it needs to be interactive (click handlers, form state, browser APIs), make it a Client Component with "use client" -- but ensure the important content is rendered by a Server Component parent.

// Server Component - AI crawlers see this content
export default async function ProductPage({ params }) {
  const product = await getProduct(params.id);

  return (
    <main>
      <h1>{product.name}</h1>
      <p>{product.description}</p>
      <p>Price: {product.price}</p>

      {/* Client Component for interactivity */}
      <AddToCartButton productId={product.id} />
    </main>
  );
}

// components/AddToCartButton.tsx
"use client";
export function AddToCartButton({ productId }) {
  // Interactive logic - AI crawlers see the button HTML
  // but the click handler only works after hydration
  return <button onClick={() => addToCart(productId)}>Add to Cart</button>;
}

Migrating from CSR to SSR

If your site is currently a client-rendered SPA and you need to make it visible to AI crawlers, here are your options in order of effort:

Option 1: Adopt a framework with SSR (recommended)

Migrate to Next.js, Nuxt, SvelteKit, or Remix. These frameworks handle SSR, SSG, and ISR out of the box. This is the most sustainable long-term solution.

Effort: High initially, but pays dividends in performance, SEO, and AI visibility. For React apps, migrating to Next.js is the most common path.

Option 2: Pre-rendering service

Services like Prerender.io or Rendertron generate static HTML snapshots of your SPA pages. When a crawler requests a page, the middleware serves the pre-rendered HTML instead of the SPA shell. Users still get the SPA experience.

Effort: Low -- usually a middleware addition. But adds latency, cost, and a dependency on a third-party service. The pre-rendered snapshots can also become stale.

Option 3: Hybrid approach

Keep your SPA for authenticated pages (dashboard, settings) but serve public-facing pages (homepage, pricing, blog) as server-rendered or static HTML. This is often the most pragmatic approach for existing applications.

// next.config.js - Example hybrid approach
module.exports = {
  async rewrites() {
    return [
      // Serve the SPA for authenticated routes
      {
        source: '/dashboard/:path*',
        destination: '/spa/index.html',
      },
      // All other routes use Next.js SSR/SSG
    ];
  },
};

Common pitfalls

Dynamic imports without SSR fallback

Using dynamic() or lazy() to code-split components can accidentally make content client-only:

// BAD: ssr: false makes this invisible to crawlers
const ProductList = dynamic(() => import('./ProductList'), {
  ssr: false,
});

// GOOD: Default SSR rendering (or explicitly set ssr: true)
const ProductList = dynamic(() => import('./ProductList'));

Content behind authentication

If your content pages require authentication (redirecting to login if not logged in), AI crawlers will be redirected and never see your content. Ensure public-facing pages are accessible without authentication.

Content loaded via useEffect

Data fetched in useEffect hooks only runs in the browser. Content loaded this way is invisible to crawlers:

// BAD: Content only loads client-side
function ProductPage() {
  const [product, setProduct] = useState(null);
  useEffect(() => {
    fetch('/api/product/123').then(r => r.json()).then(setProduct);
  }, []);
  if (!product) return <div>Loading...</div>;
  return <h1>{product.name}</h1>;
}

// GOOD: Data fetched on the server (Next.js App Router)
async function ProductPage({ params }) {
  const product = await getProduct(params.id);
  return <h1>{product.name}</h1>;
}

Loading spinners as placeholders

If AI crawlers see "Loading..." or a spinner animation instead of content, your rendering strategy needs adjustment. Every piece of content that should be visible to AI needs to be in the server-rendered HTML.

Verifying your fix

After implementing SSR, verify that AI crawlers can see your content:

# Check that content is in the HTML response
curl -s https://yourdomain.com | grep -c "<h1>"
curl -s https://yourdomain.com | grep -c "<p>"

# Verify specific content appears
curl -s https://yourdomain.com | grep "Your Product Name"

# Check multiple pages
for page in "" "/about" "/pricing" "/blog"; do
  echo "=== $page ==="
  curl -s "https://yourdomain.com$page" | wc -c
done

A well-rendered page will have thousands of bytes of HTML content. An empty SPA shell will have a few hundred bytes. The byte count is a quick sanity check.

For a comprehensive audit, use the AgentReady scanner which checks SSR status alongside all other AI readiness factors: robots.txt configuration, structured data, meta tags, llms.txt, and more.

Frequently Asked Questions

Why can't AI crawlers see my SPA content?

AI crawlers like GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript. When they fetch a single-page application (SPA), they receive the initial HTML shell which typically contains only a root div element and script tags. All the actual content is rendered client-side by JavaScript, which the crawler never runs. The result is that your content is completely invisible to AI models and will not appear in AI-generated responses.

What is the difference between SSR, SSG, and ISR?

SSR (Server-Side Rendering) generates HTML on every request at the server, ensuring content is always fresh. SSG (Static Site Generation) generates HTML once at build time, producing the fastest possible response times. ISR (Incremental Static Regeneration) generates HTML at build time but revalidates and regenerates pages in the background after a configured interval. All three produce HTML that AI crawlers can read. The choice depends on how frequently your content changes.

Does Googlebot execute JavaScript?

Yes, Googlebot uses a headless Chrome browser to render JavaScript and can see client-side content. However, AI crawlers from OpenAI (GPTBot), Anthropic (ClaudeBot), Perplexity (PerplexityBot), and others do not execute JavaScript. If you only optimise for Googlebot, your content will be invisible to the growing AI search ecosystem that drives an increasing share of web traffic.

How do I test what AI crawlers see on my site?

Use curl to fetch your pages without JavaScript execution: curl -s https://yourdomain.com | head -50. Check if the response contains your actual content or just an empty div element with script tags. You can also use "View Page Source" in Chrome (not Inspect Element) to see the raw server response. The AgentReady scanner automates this check as part of a comprehensive AI readiness audit.

Can I use pre-rendering instead of SSR?

Yes. Pre-rendering generates static HTML for each page at build time (SSG) or on first request (ISR). For content that does not change frequently, pre-rendering is more performant than SSR because it serves cached HTML instead of generating it on every request. Next.js, Nuxt, and SvelteKit all support pre-rendering alongside SSR, and you can choose the strategy per route.

Do React Server Components help with AI crawler visibility?

Yes. React Server Components (RSC) render on the server by default and send HTML to the client, not JavaScript. In Next.js App Router, all components are Server Components unless you add the "use client" directive. This means your content is in the initial HTML response, which AI crawlers can read without JavaScript execution. RSC is the most straightforward path to AI-visible React applications.

What about hydration and interactive elements?

Hydration is the process where client-side JavaScript takes over server-rendered HTML to add interactivity (click handlers, form state, animations). AI crawlers only see the initial server-rendered HTML before hydration occurs. This is fine because the text content they need is already present in the HTML. Interactive elements like buttons and forms will not function for crawlers, but the content around them is fully accessible.

Which frameworks support SSR out of the box?

Next.js (React), Nuxt (Vue), SvelteKit (Svelte), Remix (React), Astro (multi-framework), and Qwik all support server-side rendering out of the box. Next.js App Router defaults to SSR via React Server Components. Nuxt 3 defaults to universal rendering. SvelteKit defaults to SSR with per-route configuration. These are the recommended frameworks for building AI-visible websites.