By Paul GosnellSitemaps have been a core part of SEO since Google, Yahoo, and Microsoft jointly agreed on the sitemap protocol in 2006. The protocol has not changed much since then, but the way crawlers use sitemaps has evolved significantly with the rise of AI crawlers.
Traditional search engines like Google use sitemaps primarily as a discovery mechanism -- a way to find pages that internal linking might not reach. AI crawlers use sitemaps for the same purpose, but with even more reliance on the sitemap for efficient content discovery. AI crawlers tend to be more targeted than broad search crawlers, focusing on content that is relevant to their specific use case. A well-structured sitemap helps them find the right content quickly.
This guide covers the XML sitemap format, the Next.js sitemap.ts pattern, dynamic versus static approaches, sitemap indexes for large sites, and how to submit your sitemap to Google Search Console. Use the AgentReady scanner to check if your sitemap is properly configured.
XML Sitemap Format
An XML sitemap follows a strict schema defined by the Sitemaps protocol (sitemaps.org). Here is the basic structure.
Minimal sitemap
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://yourdomain.com/</loc>
</url>
<url>
<loc>https://yourdomain.com/about</loc>
</url>
<url>
<loc>https://yourdomain.com/pricing</loc>
</url>
</urlset>Full sitemap with metadata
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://yourdomain.com/</loc>
<lastmod>2026-03-12</lastmod>
<changefreq>weekly</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://yourdomain.com/pricing</loc>
<lastmod>2026-03-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>https://yourdomain.com/blog/ai-agents-guide</loc>
<lastmod>2026-02-15</lastmod>
<changefreq>monthly</changefreq>
<priority>0.6</priority>
</url>
</urlset>Element reference
| Element | Required | Description |
|---|---|---|
<urlset> | Yes | Root element with namespace declaration |
<url> | Yes | Container for each URL entry |
<loc> | Yes | Full URL of the page (must include protocol) |
<lastmod> | No | Last modification date (W3C datetime format) |
<changefreq> | No | How often the page changes (always, hourly, daily, weekly, monthly, yearly, never) |
<priority> | No | Relative priority from 0.0 to 1.0 (default: 0.5) |
Important rules
- All URLs must be fully qualified with protocol (https://)
- URLs must be from the same domain as the sitemap
- Maximum 50,000 URLs per sitemap file
- Maximum 50MB uncompressed file size
- Special characters in URLs must be entity-escaped (
&,', etc.) - Only include canonical URLs (no duplicate content, no redirects)
- Only include pages that return 200 status codes
Priority and changefreq: Do They Matter?
Google has publicly stated that it ignores both priority and changefreq values in sitemaps. Google relies on its own crawl data to determine how often to revisit pages. However, other search engines and some AI crawlers may still use these signals.
The most useful optional element is lastmod. Accurate last modification dates help crawlers prioritise which pages to re-crawl. If you update a page, updating its lastmod value signals to crawlers that there is new content to index.
Our recommendation: always include lastmod with accurate dates. Optionally include priority and changefreq as they do not hurt and provide additional signals for non-Google crawlers.
Next.js Sitemap Implementation
Option 1: Static sitemap in public/
For simple sites with a fixed set of pages, create public/sitemap.xml manually. This file is served at /sitemap.xml automatically.
Option 2: Dynamic sitemap.ts (App Router)
Next.js App Router supports a sitemap.ts file in the app/ directory that generates the sitemap at build time. This is the recommended approach for most sites.
// app/sitemap.ts
import { MetadataRoute } from 'next'
export default function sitemap(): MetadataRoute.Sitemap {
const baseUrl = 'https://yourdomain.com'
// Static pages
const staticPages: MetadataRoute.Sitemap = [
{
url: baseUrl,
lastModified: new Date(),
changeFrequency: 'weekly',
priority: 1,
},
{
url: `${baseUrl}/about`,
lastModified: new Date(),
changeFrequency: 'monthly',
priority: 0.8,
},
{
url: `${baseUrl}/pricing`,
lastModified: new Date(),
changeFrequency: 'monthly',
priority: 0.8,
},
{
url: `${baseUrl}/contact`,
lastModified: new Date(),
changeFrequency: 'yearly',
priority: 0.5,
},
]
return staticPages
}Dynamic sitemap with database content
For sites with dynamic content (blog posts, products, user-generated pages), fetch the URLs from your database at build time.
// app/sitemap.ts
import { MetadataRoute } from 'next'
import { createClient } from '@/lib/supabase/server'
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
const baseUrl = 'https://yourdomain.com'
const supabase = await createClient()
// Fetch blog posts from database
const { data: posts } = await supabase
.from('posts')
.select('slug, updated_at')
.eq('status', 'published')
.order('updated_at', { ascending: false })
const blogUrls: MetadataRoute.Sitemap = (posts || []).map((post) => ({
url: `${baseUrl}/blog/${post.slug}`,
lastModified: new Date(post.updated_at),
changeFrequency: 'monthly' as const,
priority: 0.6,
}))
// Static pages
const staticPages: MetadataRoute.Sitemap = [
{ url: baseUrl, lastModified: new Date(), priority: 1 },
{ url: `${baseUrl}/about`, lastModified: new Date(), priority: 0.8 },
{ url: `${baseUrl}/pricing`, lastModified: new Date(), priority: 0.8 },
{ url: `${baseUrl}/blog`, lastModified: new Date(), priority: 0.7 },
]
return [...staticPages, ...blogUrls]
}Including static HTML pages from public/
If your site has static HTML pages in the public/ directory (common for SEO content pages), you need to include them in your sitemap. Since Next.js does not automatically discover files in public/, you have two options:
// Option A: Hardcode the paths
const guidePaths = [
'/guides/robots-txt-guide.html',
'/guides/ai-crawlers-guide.html',
'/guides/llms-txt-guide.html',
'/guides/sitemap-guide.html',
'/guides/mcp-server-guide.html',
]
const guideUrls = guidePaths.map(path => ({
url: `${baseUrl}${path}`,
lastModified: new Date('2026-03-12'),
priority: 0.7,
}))
// Option B: Read the filesystem at build time
import { readdirSync } from 'fs'
import { join } from 'path'
const guidesDir = join(process.cwd(), 'public', 'guides')
const guideFiles = readdirSync(guidesDir)
.filter(f => f.endsWith('.html'))
const guideUrls = guideFiles.map(file => ({
url: `${baseUrl}/guides/${file}`,
lastModified: new Date(),
priority: 0.7,
}))Sitemap Index for Large Sites
When your site has more than 50,000 URLs or you want to organise URLs by type, use a sitemap index. A sitemap index is an XML file that references multiple individual sitemap files.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://yourdomain.com/sitemap-pages.xml</loc>
<lastmod>2026-03-12</lastmod>
</sitemap>
<sitemap>
<loc>https://yourdomain.com/sitemap-blog.xml</loc>
<lastmod>2026-03-10</lastmod>
</sitemap>
<sitemap>
<loc>https://yourdomain.com/sitemap-guides.xml</loc>
<lastmod>2026-03-12</lastmod>
</sitemap>
</sitemapindex>In Next.js, you can generate multiple sitemaps using the generateSitemaps function:
// app/sitemap.ts
import { MetadataRoute } from 'next'
export async function generateSitemaps() {
// Return an array of sitemap IDs
return [
{ id: 'pages' },
{ id: 'blog' },
{ id: 'guides' },
]
}
export default async function sitemap({
id,
}: {
id: string
}): Promise<MetadataRoute.Sitemap> {
const baseUrl = 'https://yourdomain.com'
if (id === 'pages') {
return [
{ url: baseUrl, lastModified: new Date(), priority: 1 },
{ url: `${baseUrl}/about`, lastModified: new Date(), priority: 0.8 },
]
}
if (id === 'blog') {
// Fetch blog posts from database
return [
{ url: `${baseUrl}/blog/post-1`, lastModified: new Date() },
]
}
// guides
return [
{ url: `${baseUrl}/guides/robots-txt-guide.html`, lastModified: new Date() },
]
}Submitting Your Sitemap to Google Search Console
Step 1: Verify site ownership
Before you can submit a sitemap, you need to verify ownership of your site in Google Search Console. If you have not done this yet, go to search.google.com/search-console, add your property, and verify using DNS, HTML tag, or file upload.
Step 2: Submit the sitemap
- Open Google Search Console
- Select your property
- Click "Sitemaps" in the left sidebar
- Enter your sitemap URL (e.g.,
sitemap.xml) - Click "Submit"
Step 3: Monitor for errors
After submission, Google will crawl your sitemap and report any issues. Common errors include:
- Invalid XML: Check for unclosed tags or invalid characters
- URLs returning 404: Remove deleted pages from your sitemap
- URLs blocked by robots.txt: Do not include URLs that your robots.txt blocks
- Redirect chains: Only include the final destination URL, not redirected URLs
- Non-canonical URLs: Only include canonical versions of pages
Reference in robots.txt
Always add a Sitemap: directive to your robots.txt file. This ensures all crawlers (not just those you manually submit to) can find your sitemap.
# At the bottom of robots.txt
Sitemap: https://yourdomain.com/sitemap.xmlSitemaps and AI Crawlers
AI crawlers use sitemaps in several ways that differ from traditional search engines:
Content discovery
AI crawlers like GPTBot and PerplexityBot use sitemaps to discover content efficiently. Rather than following every link on your site, they can fetch your sitemap to get a complete list of URLs and prioritise which pages to crawl based on lastmod dates and content type.
Freshness signals
The lastmod element is particularly important for AI crawlers that power real-time search features. When PerplexityBot or OAI-SearchBot sees that a page was recently modified, it prioritises re-crawling that page to ensure its search index has the latest information.
Scope understanding
The sitemap gives AI crawlers a sense of your site's scope and structure. A site with 500 pages in its sitemap is understood differently from a site with 5 pages. This context helps AI systems calibrate how much they rely on your site as a source.
Best Practices
- Only include indexable pages: No 404s, no redirects, no noindex pages
- Use absolute URLs: Always include the full URL with protocol and domain
- Keep lastmod accurate: Only update lastmod when content actually changes
- One canonical URL per page: Do not include both
wwwand non-wwwversions - Gzip for large sitemaps: Serve
sitemap.xml.gzif your sitemap is over 10MB - Reference in robots.txt: Always include the Sitemap directive
- Update on deploy: Regenerate your sitemap whenever you deploy new content
- Include all content types: Static pages, blog posts, guide pages, product pages
Sitemaps in the AI Readiness Stack
Your sitemap works alongside other AI readiness components:
- robots.txt -- controls which crawlers can access your site and references your sitemap
- llms.txt -- provides context about what your site does (the sitemap lists where things are)
- AI crawler rules -- determine which AI bots can use your sitemap to discover content
- MCP server -- enables programmatic interaction beyond passive crawling
Check all of these with the AgentReady scanner.
Frequently Asked Questions
What is an XML sitemap?
An XML sitemap is a file that lists all the important URLs on your website, along with metadata like last modification date and change frequency. It helps search engines and AI crawlers discover and prioritise your content for indexing, especially pages that may not be easily found through internal linking alone.
Do AI crawlers use sitemaps?
Yes. AI crawlers like GPTBot, PerplexityBot, and OAI-SearchBot use sitemaps to discover content efficiently. The Sitemap directive in robots.txt points crawlers to your sitemap, and many AI crawlers check for /sitemap.xml automatically even without a robots.txt reference.
How do I create a sitemap in Next.js?
Next.js App Router supports a sitemap.ts file in the app/ directory that exports a function returning sitemap entries. This generates sitemap.xml dynamically at build time. You can also place a static sitemap.xml in the public/ directory for simple sites.
What is a sitemap index?
A sitemap index is an XML file that references multiple sitemap files. It is used when your site has more than 50,000 URLs (the maximum per sitemap) or when you want to organise URLs by type (pages, blog posts, images). Search engines and AI crawlers follow the references to find all individual sitemaps.
Should I include priority and changefreq in my sitemap?
Google has stated it ignores priority and changefreq values. However, other crawlers and AI bots may still use them as hints. Including them does not hurt and provides additional signals for non-Google crawlers. Focus on accurate lastmod dates as the most universally useful metadata element.
How often should I update my sitemap?
Your sitemap should update automatically whenever content changes. For dynamic Next.js sites, the sitemap.ts approach regenerates on each build or deployment. The lastmod dates should reflect actual content changes, not arbitrary timestamps -- inaccurate dates reduce crawler trust in your sitemap.
How do I submit my sitemap to Google Search Console?
In Google Search Console, go to Sitemaps in the left sidebar, enter your sitemap URL (e.g., https://yourdomain.com/sitemap.xml), and click Submit. Google will crawl it and report any errors. You can also reference it in your robots.txt with the Sitemap: directive for automatic discovery by all crawlers.
What is the maximum size for an XML sitemap?
A single XML sitemap can contain a maximum of 50,000 URLs and must not exceed 50MB uncompressed. If your site exceeds these limits, use a sitemap index that references multiple sitemap files. You can also gzip your sitemaps to reduce transfer size.