How to Create a robots.txt File for AI Crawlers

The robots.txt file has been a cornerstone of web crawling etiquette since 1994, when it was introduced as a simple way for webmasters to communicate with search engine bots. For decades, the conversation was straightforward: allow or block Googlebot, Bingbot, and a handful of other search crawlers.

That changed dramatically in 2023 when OpenAI, Anthropic, Perplexity, and other AI companies began deploying their own web crawlers to gather training data and power real-time search features. Suddenly, the robots.txt file became the frontline of a much larger conversation about AI access, content licensing, and digital visibility.

Today, your robots.txt configuration directly determines whether your content appears in ChatGPT responses, Claude conversations, Perplexity search results, Google AI Overviews, and dozens of other AI-powered surfaces. Getting it right is no longer optional for any business that relies on organic traffic.

This guide covers everything you need to know: the syntax, every major AI crawler user-agent, complete configuration examples, common mistakes to avoid, and how to implement robots.txt in modern frameworks like Next.js. Use our AgentReady scanner to audit your current robots.txt configuration automatically.

What is robots.txt?

robots.txt is a plain text file placed at the root of your website (accessible at https://yourdomain.com/robots.txt) that provides instructions to web crawlers about which parts of your site they can access. It follows the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers honour.

The key word is voluntary. robots.txt is not a security mechanism. It does not enforce access controls. It is a polite request that legitimate crawlers respect. Malicious bots and scrapers will ignore it entirely. But every major AI crawler from OpenAI, Anthropic, Google, Microsoft, and Perplexity respects robots.txt rules.

How AI crawlers use robots.txt

When an AI crawler visits your website, the first thing it does is fetch /robots.txt. It looks for rules that match its user-agent string. If it finds a Disallow directive for its user-agent, it will not crawl the specified paths. If it finds Allow directives, or no rules at all, it proceeds to crawl your content.

AI crawlers use your content for several purposes depending on the company and the specific crawler:

• Training data: Content is ingested into large language model training datasets. This is what GPTBot primarily does.
• Real-time search: Content is fetched live when a user asks a question. ChatGPT-User and PerplexityBot do this.
• Search indexing: Content is indexed for AI-powered search features like Google AI Overview and Bing Copilot.
• Grounding and retrieval: Content is used to verify and ground AI responses with factual, up-to-date information.

robots.txt Syntax: The Complete Reference

The robots.txt format is simple but has several directives you need to understand. Here is the complete syntax reference.

User-agent directive

The User-agent directive specifies which crawler the following rules apply to. Use * to target all crawlers, or specify individual bot names for granular control.

# Apply to all crawlers
User-agent: *
Disallow: /admin/

# Apply only to GPTBot
User-agent: GPTBot
Allow: /

Allow and Disallow directives

The Disallow directive blocks access to specific paths. The Allow directive explicitly permits access, overriding broader Disallow rules. An empty Disallow: means nothing is blocked.

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Allow: /api/public/

Sitemap directive

The Sitemap directive tells crawlers where to find your XML sitemap. This is especially important for AI crawlers that use sitemaps to discover content efficiently. See our XML Sitemap Guide for more details.

Sitemap: https://yourdomain.com/sitemap.xml

Crawl-delay directive

The Crawl-delay directive asks crawlers to wait a specified number of seconds between requests. Not all crawlers respect this (Google ignores it), but some AI crawlers do honour it.

User-agent: GPTBot
Crawl-delay: 2

Wildcard patterns

Many crawlers support wildcard patterns using * to match any sequence of characters and $ to match the end of a URL.

# Block all PDF files
User-agent: *
Disallow: /*.pdf$

# Block URLs containing "print"
User-agent: *
Disallow: /*print*

Comments

Lines starting with # are comments and are ignored by crawlers. Use comments generously to document your configuration.

# This is a comment explaining the next rule
User-agent: GPTBot
Allow: / # Allow full access for ChatGPT search

All Major AI Crawler User-Agents

Here is a comprehensive reference of every AI crawler you should consider in your robots.txt configuration. For a deeper dive into each crawler, see our Complete Guide to AI Crawlers.

User-Agent	Company	Purpose
GPTBot	OpenAI	Training data and search indexing
ChatGPT-User	OpenAI	Real-time web browsing in ChatGPT
OAI-SearchBot	OpenAI	SearchGPT / ChatGPT search results
ClaudeBot	Anthropic	Training data for Claude models
anthropic-ai	Anthropic	General Anthropic crawler
PerplexityBot	Perplexity	Real-time search and answer generation
Google-Extended	Google	Gemini training data (separate from search)
Applebot-Extended	Apple	Apple Intelligence and Siri features
CCBot	Common Crawl	Open dataset used by many AI labs
Amazonbot	Amazon	Alexa and Amazon search features
Meta-ExternalAgent	Meta	Meta AI training and features
Bytespider	ByteDance	TikTok and ByteDance AI features
Diffbot	Diffbot	Structured data extraction for AI
cohere-ai	Cohere	Cohere model training data
YouBot	You.com	You.com AI search engine

Complete robots.txt Examples

Maximum AI visibility (recommended for most sites)

This configuration allows all AI crawlers full access to your public content while blocking admin and API paths. This is the recommended starting point for any business that wants maximum visibility in AI search results.

# ===========================================
# robots.txt - Maximum AI Visibility
# ===========================================

# Default: allow all crawlers
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/
Disallow: /private/
Disallow: /_next/

# Explicitly allow AI crawlers (for clarity)
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: Bytespider
Allow: /

User-agent: Diffbot
Allow: /

User-agent: YouBot
Allow: /

# Allow AI agent endpoints
User-agent: *
Allow: /api/mcp
Allow: /api/agent
Allow: /api/ai/context
Allow: /.well-known/agent.json

# Sitemap
Sitemap: https://yourdomain.com/sitemap.xml

Selective AI access (allow search, block training)

Some sites want to appear in AI search results but do not want their content used for model training. This configuration allows real-time search crawlers while blocking training-focused ones.

# ===========================================
# robots.txt - Allow AI Search, Block Training
# ===========================================

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

# Allow real-time search crawlers
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Block training-focused crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: cohere-ai
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Block all AI crawlers

If you have specific reasons to block all AI access (paywalled content, licensing restrictions, etc.), this configuration blocks every known AI crawler while still allowing traditional search engines.

# ===========================================
# robots.txt - Block All AI Crawlers
# ===========================================

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Block all AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: YouBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Common Mistakes to Avoid

Mistake 1: Blocking all bots with a wildcard

The most common and damaging mistake is using a blanket disallow that blocks all crawlers, including AI ones:

# BAD - blocks everything including AI crawlers
User-agent: *
Disallow: /

This prevents your content from appearing in any AI search results. If you see this in your robots.txt, it is almost certainly too restrictive. Use specific disallow paths instead of blocking the root.

Mistake 2: Not having a robots.txt file at all

If your site has no robots.txt file, crawlers will access everything by default. While this means AI crawlers can access your content, you also lose control over which paths they crawl. Admin panels, API endpoints, and staging content may all be indexed.

Mistake 3: Placing robots.txt in the wrong location

robots.txt must be at the root of your domain: https://yourdomain.com/robots.txt. Placing it at /public/robots.txt or in a subdirectory means crawlers will never find it. In Next.js, put it in the public/ directory so it is served at the root.

Mistake 4: Forgetting about subdomains

Each subdomain needs its own robots.txt. The file at www.yourdomain.com/robots.txt does not apply to app.yourdomain.com or blog.yourdomain.com. Make sure every subdomain that serves content has its own properly configured robots.txt.

Mistake 5: Using robots.txt for security

Never rely on robots.txt to protect sensitive content. It is a public file that anyone can read, and it only works with compliant crawlers. Use authentication, access controls, and server-side restrictions for actual security.

Mistake 6: Blocking JavaScript and CSS files

Some older robots.txt configurations block *.js and *.css files. Modern crawlers (including AI ones) need access to these resources to properly render and understand your pages. Never block these file types.

Mistake 7: Not including a Sitemap directive

The Sitemap: directive is free metadata that helps crawlers discover your content more efficiently. Always include it, pointing to your XML sitemap. If you have multiple sitemaps, include multiple Sitemap: lines.

Implementing robots.txt in Next.js

Option 1: Static file in public/

The simplest approach is to place a robots.txt file directly in your public/ directory. This file will be served at /robots.txt automatically.

# public/robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /_next/

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Option 2: Dynamic robots.ts (App Router)

Next.js App Router supports a robots.ts file in the app/ directory that generates robots.txt dynamically. This lets you use environment variables to serve different configurations for production and staging.

// app/robots.ts
import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://yourdomain.com'

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin/', '/api/', '/_next/', '/dashboard/'],
      },
      {
        userAgent: 'GPTBot',
        allow: '/',
      },
      {
        userAgent: 'ChatGPT-User',
        allow: '/',
      },
      {
        userAgent: 'OAI-SearchBot',
        allow: '/',
      },
      {
        userAgent: 'ClaudeBot',
        allow: '/',
      },
      {
        userAgent: 'anthropic-ai',
        allow: '/',
      },
      {
        userAgent: 'PerplexityBot',
        allow: '/',
      },
      {
        userAgent: 'Google-Extended',
        allow: '/',
      },
      {
        userAgent: 'Applebot-Extended',
        allow: '/',
      },
      {
        userAgent: 'Meta-ExternalAgent',
        allow: '/',
      },
      {
        userAgent: 'CCBot',
        allow: '/',
      },
      {
        userAgent: 'Amazonbot',
        allow: '/',
      },
      {
        userAgent: 'cohere-ai',
        allow: '/',
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  }
}

The advantage of the dynamic approach is that you can serve different rules per environment. For example, you might block all crawlers on staging domains:

// app/robots.ts
import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.NODE_ENV === 'production'
    && process.env.VERCEL_ENV === 'production'

  if (!isProduction) {
    return {
      rules: { userAgent: '*', disallow: '/' },
    }
  }

  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/admin/', '/api/'] },
      { userAgent: 'GPTBot', allow: '/' },
      { userAgent: 'ClaudeBot', allow: '/' },
      { userAgent: 'PerplexityBot', allow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Testing Your robots.txt Configuration

Google Search Console

Google Search Console provides a robots.txt Tester tool that lets you validate your syntax and test specific URLs against your rules. Navigate to Crawl > robots.txt Tester in the old Search Console interface, or use the URL Inspection tool in the new interface to see how Googlebot interprets your rules.

Online validators

Several free tools can validate your robots.txt syntax:

• Google robots.txt Tester (via Search Console) - the most authoritative option
• Bing Webmaster Tools - validate against Bingbot rules
• robotstxt.org - independent syntax validation
• Screaming Frog - crawl your site and verify robots.txt is respected

Manual testing

The simplest test is to fetch your robots.txt directly in a browser:

curl https://yourdomain.com/robots.txt

Verify that:

1. The file is accessible (returns 200, not 404)
2. The content type is text/plain
3. AI crawler user-agents have explicit Allow rules
4. The Sitemap directive points to a valid URL
5. Admin and private paths are blocked
6. There are no wildcard Disallow rules that inadvertently block AI crawlers

AgentReady scanner

Use the p0stman AgentReady scanner to automatically audit your robots.txt alongside your entire AI-readiness configuration. It checks for missing AI crawler rules, incorrect syntax, and opportunities to improve your AI search visibility.

robots.txt and the Broader AI Readiness Stack

robots.txt is just one component of making your website visible to AI systems. For comprehensive AI readiness, you also need:

• llms.txt: A plain-text summary of your site specifically for language models, providing structured context about your business, products, and API capabilities.
• XML Sitemap: Helps AI crawlers discover all your content efficiently, especially important for large sites with deep page structures.
• MCP Server: Enables AI agents to programmatically interact with your product through the Model Context Protocol, going beyond passive crawling to active tool use.
• Structured data (JSON-LD): Schema markup helps AI systems understand the relationships between your content entities, improving citation accuracy.
• Agent discovery files: mcp.json and .well-known/agent.json enable AI agents to discover your API capabilities automatically.

Run a comprehensive audit of all these components with the AgentReady scanner.

The Business Case for AI Crawler Access

Some businesses hesitate to allow AI crawlers, concerned about content being used for training without compensation. Here is the data-driven case for allowing access.

AI referral traffic is growing exponentially

According to multiple analytics studies through 2025-2026, referral traffic from AI sources (ChatGPT, Perplexity, Claude) is growing at 300-500% year-over-year for sites that are properly indexed. Blocking AI crawlers means forfeiting this traffic entirely.

AI visitors convert better than organic search

Adobe's 2025 data shows that visitors arriving from AI referrals convert at 4.4x the rate of organic search visitors. These users arrive with high intent and specific context. They have already asked a question and received a recommendation to visit your site.

AI visibility compounds over time

When AI models cite your content, they create a self-reinforcing loop. Users trust the recommendation, visit your site, engage with your content, and the AI learns that your site is a valuable source. This compounds over time, similar to traditional SEO authority building but faster.

The cost of blocking is invisible

The traffic you lose by blocking AI crawlers never shows up in your analytics. You cannot see what you are missing. But competitors who allow access are building AI visibility while you are not. The longer you block, the harder it becomes to catch up.

Frequently Asked Questions

What is robots.txt and why does it matter for AI crawlers?

robots.txt is a plain text file at the root of your website that tells web crawlers which pages they can and cannot access. For AI crawlers like GPTBot, ClaudeBot, and PerplexityBot, it determines whether your content appears in AI-generated responses, ChatGPT search results, and AI overviews. Without proper configuration, your site may be invisible to the fastest-growing traffic sources on the web.

Where should robots.txt be placed?

robots.txt must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. It cannot be in a subdirectory. For Next.js projects, place it in the public/ directory or use the app/robots.ts dynamic generation approach. Each subdomain needs its own robots.txt file.

Does blocking AI crawlers in robots.txt remove my content from AI responses?

Blocking AI crawlers prevents them from accessing your content going forward, but it does not retroactively remove content that was already crawled and indexed. If your content was previously crawled, it may still appear in AI responses until the model is retrained without it. OpenAI provides a separate opt-out form for removing content from training datasets.

Should I allow or block GPTBot on my website?

For most businesses, allowing GPTBot is beneficial. It enables your content to appear in ChatGPT responses and OpenAI search results, driving referral traffic that converts at 4.4x the rate of organic search. Block GPTBot only if you have specific concerns about content licensing or if your content is behind a paywall.

What is the difference between GPTBot and ChatGPT-User?

GPTBot crawls content for training and search indexing across OpenAI products. ChatGPT-User is the user-agent used when ChatGPT browses the web in real-time during a conversation. You can allow ChatGPT-User for live browsing while blocking GPTBot for training, giving you granular control over how your content is used.

How do I test my robots.txt file?

Use Google Search Console's robots.txt Tester to validate syntax and test specific URLs against your rules. You can also use online validators like robotstxt.org, or simply fetch https://yourdomain.com/robots.txt in your browser to verify it is accessible and correctly formatted. The AgentReady scanner automates this check as part of a broader AI readiness audit.

Can I use robots.txt to allow AI crawlers but block traditional search engines?

Yes. robots.txt supports per-user-agent rules. You can create separate blocks for Googlebot, Bingbot, GPTBot, ClaudeBot, and others, each with different Allow and Disallow directives. This gives you granular control over which crawlers can access which content.

How does robots.txt work with Next.js App Router?

Next.js App Router supports a robots.ts file in the app/ directory that exports a metadata function. This generates robots.txt dynamically at build time, letting you use TypeScript and environment variables to configure crawler rules programmatically. This is ideal for serving different rules on staging versus production environments.