What are the 5 files every agent-ready website needs?

The five files are: (1) llms.txt: the AI index file that tells LLMs who you are; (2) robots.txt: updated with explicit permissions for AI crawlers like GPTBot and ClaudeBot; (3) /.well-known/agent.json: the AgentCard for A2A agent discovery per RFC 8615; (4) mcp.json: the MCP tool registry listing actions AI agents can take on your site; (5) agents.md: the capability manifest giving agents detailed operational context.

Does my site need to be rebuilt in Next.js to support these files?

No. All five files work on any stack including WordPress, Webflow, Squarespace, or raw HTML. llms.txt, robots.txt, mcp.json, and agents.md are plain text or JSON served at fixed paths. The MCP server and A2A endpoint require a server-side runtime but can be deployed as standalone APIs alongside any existing site.

What is agent.json and why is it served at /.well-known/?

agent.json is the AgentCard: a machine-readable JSON file that declares your AI agent's identity, capabilities, and task endpoint for the A2A (Agent-to-Agent) protocol. It is served at the /.well-known/ path because RFC 8615 defines this as the standard location for well-known URIs. Any A2A-compatible system, backed by the Linux Foundation with 150+ organisations including Google, Microsoft, and IBM, knows to look there for agent discovery.

How quickly will AI systems discover these files after I create them?

GPTBot and Google-Extended typically crawl within 1–3 days. For faster indexing, submit your URLs via IndexNow: Bing's real-time indexing protocol that also signals Yandex and other participants. For MCP and A2A, there is no crawl delay: agents call these endpoints in real time as needed, so they are effective from the moment they are deployed.

p0.

Work Services Process About Contact

AI VisibilityAre AI engines citing you?AgentReadyAI readiness audit

Agentic Web/Five files

Agent-Ready Files

The five filesevery site needs.

Q: What is the difference between llms.txt and agents.md?

llms.txt is the broad overview for passive citation: it tells any LLM what your business is, what you offer, and how to reach you. agents.md is the operational guide for active agents: it explains how to call your endpoints, what tasks work well, what to expect in responses, and known limitations. Think of llms.txt as your press kit and agents.md as your API documentation written in plain English.

By 2027, every agent-ready site will expose five machine-readable files: llms.txt, robots.txt, an AgentCard, an MCP manifest, and agents.md. All five let AI agents discover, understand, and act, without a human touching a browser.

Live on p0stman.com now

Five files, all live

From £1,500

Scroll

Build this for my site See the full four-layer build

The Shift

Thirty years of web design assumed a human was doing the clicking. That assumption is already breaking down.

AI agents (Claude, ChatGPT, Gemini, LangGraph orchestrators, AutoGPT successors) are now among the fastest-growing categories of web traffic. They don't use browsers. They don't click links. They read structured data, call APIs, and take actions on behalf of users.

A website that doesn't expose the right files to these agents is invisible to them. Not penalised. Just absent. The agent finds a competitor with the infrastructure instead, and the human never knows your site existed.

These are the five files that separate agent-ready websites from the rest. All five are live on p0stman.com: every link on this page goes to the real file.

The Five Files

What each file does, and how to build it.

Layer 1 · Discovery

llms.txt

The AI index file

A plain-text file at the root of your domain that tells large language models who you are, what you do, and where to find key information about your business.

When ChatGPT, Claude, or Perplexity decides how to describe your business to a user, it needs somewhere to start. Without llms.txt, it guesses from scraped HTML: getting your pricing wrong, misrepresenting your services, or confusing you with a competitor.

llms.txt gives you direct input into that process. It's the equivalent of a press kit, written for machines. Proposed by Jeremy Howard (fast.ai) in 2024, it's now adopted by hundreds of companies and actively read by LLM crawlers.

Example

text

# p0stman

> AI-native product studio. We build voice agents, AI-powered
> applications, and custom software for businesses that want to
> move faster than traditional agencies allow.

Contact: hello@p0stman.com | https://p0stman.com
MCP server: https://p0stman.com/api/mcp

## Services

### AI Voice Agents: from £5,000 | 6-10 days
Inbound and outbound voice AI for hotels and restaurants.

### MVP Launch: from £5,000 | 1-3 weeks
From idea to live product in weeks.

How to implement

Create a file named llms.txt at the root of your site. Write in plain markdown. Include: company description, services with pricing, contact details, and key page URLs. 500–2,000 words. No HTML. Serve it at the root.

Live example: p0stman.com/llms.txt

Layer 1 · Discovery

robots.txt

The AI permissions file

The standard file that tells web crawlers what they can and cannot access, updated to explicitly allow the new generation of AI crawlers that most sites were not built to accommodate.

Most robots.txt files were written before AI crawlers existed. A blanket Disallow or an unconfigured file means AI systems from OpenAI, Anthropic, and Google may skip your site entirely, or only partially index it.

The new AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) check robots.txt before reading anything. If they're not explicitly allowed, the safest implementations will skip you. Worse, some will allow crawling but not attribute your content correctly without explicit permission signals.

Example

text

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Googlebot
Allow: /

Disallow: /admin
Disallow: /api/admin

Sitemap: https://yoursite.com/sitemap.xml

How to implement

Open your existing robots.txt and add explicit Allow rules for each AI crawler. The key bots to allow: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, TavilyBot. Keep your existing rules for admin and private areas.

Live example: p0stman.com/robots.txt

Layer 4 · Agent-to-Agent

/.well-known/agent.json

The AgentCard

A machine-readable JSON file served at the RFC 8615 well-known path that declares your AI agent's identity, capabilities, and task endpoint: the business card of the agentic web.

The A2A (Agent-to-Agent) protocol, backed by Google, Microsoft, IBM, and 150+ organisations under the Linux Foundation, defines how AI agents discover and communicate with each other. The AgentCard is the discovery mechanism.

When an AI agent (say, a customer's personal AI assistant) wants to interact with your business autonomously, it checks /.well-known/agent.json first. Without it, your business doesn't exist to that agent. With it, the agent knows your name, what you can do, and exactly how to send you a task.

The well-known path (RFC 8615) is intentional: the same convention used by ACME challenges, security.txt, and OpenID Connect. Every A2A-compatible system knows to look there.

Example

json

{
  "name": "Zero",
  "description": "AI ops assistant at p0stman",
  "version": "1.0",
  "url": "https://p0stman.com/api/agent",
  "capabilities": {
    "streaming": false,
    "pushNotifications": false
  },
  "skills": [
    {
      "id": "inquire",
      "name": "Agency Inquiry",
      "description": "Answer questions about services and pricing"
    },
    {
      "id": "book",
      "name": "Book Discovery Call",
      "description": "Schedule a free 30-minute discovery call"
    }
  ],
  "authentication": { "schemes": ["None"] },
  "provider": {
    "organization": "p0stman",
    "url": "https://p0stman.com"
  }
}

How to implement

Create a JSON file at public/.well-known/agent.json (or serve it as a route at /.well-known/agent.json). Include: name, description, version, the URL of your A2A task endpoint, capabilities, skills with IDs and descriptions, and authentication scheme. Pair it with an actual /api/agent endpoint that accepts JSON-RPC tasks/send requests.

Live example: p0stman.com/.well-known/agent.json

Layer 3 · Action

mcp.json

The MCP tool registry

A JSON manifest listing the tools (actions) your site exposes via the Model Context Protocol: the file that tells AI assistants what they can actually do on your site.

MCP (Model Context Protocol), open-sourced by Anthropic and now adopted by OpenAI, Google, and dozens of AI platforms, defines how AI agents call tools on external services. Claude Desktop, Cursor, and a growing list of AI assistants can read mcp.json and offer your tools directly to their users.

Without mcp.json and a backing MCP server, your business is read-only to AI agents. They can learn about you from llms.txt, but they can't act. With MCP, an agent can check your availability, submit an enquiry, or book a call on behalf of a user, all within their AI interface, without visiting your site.

This is where passive discovery becomes active integration.

Example

json

{
  "name": "p0stman AI Studio",
  "description": "AI-native product studio. Book calls,
                  browse services, view portfolio.",
  "version": "1.0.0",
  "endpoint": "https://p0stman.com/api/mcp",
  "protocol": "json-rpc-2.0",
  "tools": [
    {
      "name": "book_discovery_call",
      "description": "Book a free 30-minute discovery call."
    },
    {
      "name": "submit_inquiry",
      "description": "Submit a project inquiry."
    },
    {
      "name": "get_services",
      "description": "Get services with pricing and timelines."
    }
  ]
}

How to implement

Create mcp.json at your site root listing your tools with names and descriptions. Build a corresponding MCP server at /api/mcp that handles tools/list (returns all tools with inputSchema) and tools/call (executes the tool and returns structured results). Use JSON-RPC 2.0 format.

Live example: p0stman.com/mcp.json

Layer 1 · Discovery

agents.md

The capability manifest

A markdown file that bridges the gap between the brief llms.txt and the structured agent.json: giving AI agents detailed, nuanced context about what your agent can do and how to work with it effectively.

llms.txt tells LLMs who you are. agent.json tells other agents how to call you. agents.md tells agents what to actually do and what to expect: the nuance that structured JSON can't capture.

It's where you explain edge cases, preferred task formats, example queries that work well, limitations to be aware of, and capabilities not covered by the formal schema. Think of it as the README for your agent.

As multi-agent orchestration matures, agents will read agents.md before deciding whether to delegate a task to your agent, how to phrase the request, and what format to expect back. The more useful and specific it is, the more reliably other agents can work with yours.

Example

text

# p0stman: Agent Instructions

## About
AI-native product studio. Builds voice agents, MVPs,
web apps for businesses.

## MCP Server (Recommended)
Endpoint: https://p0stman.com/api/mcp
Protocol: JSON-RPC 2.0

## A2A Task Endpoint
Endpoint: https://p0stman.com/api/agent
Protocol: JSON-RPC 2.0 (tasks/send method)

## Example Tasks
- "What AI services do you offer for restaurants?"
- "What would an AI voice agent cost for a 3-site hotel?"
- "Book a discovery call for [name] at [email]"

## Limitations
- Cannot process payments directly
- Discovery calls must be confirmed by email

How to implement

Create agents.md at your site root. Include: who you are (1 paragraph), MCP endpoint with example calls, A2A endpoint with example tasks, what requests work well, known limitations, and contact fallback. Write for a machine reader: structured, specific, no marketing fluff.

Live example: p0stman.com/agents.md

The Sequence

How the five files work together.

Discover

AI crawlers find your site. LLMs learn what you do. You get correctly cited in AI responses.

llms.txt · robots.txt

Understand

Agents read the detail. They learn your capabilities, constraints, and how to phrase requests to get useful responses.

agents.md

Act

Agents call your MCP tools. They can check services, search content, and book calls, without any human touching a browser.

mcp.json

Orchestrate

Other AI agents discover Zero via the AgentCard, send tasks, and receive structured responses. Machine-to-machine business.

/.well-known/agent.json

The Cost

What happens to sites that don't have them.

Not penalised. Just absent, in the exact moment a buying decision is being made.

Misrepresented by LLMs

ChatGPT and Claude guess from scraped HTML: wrong pricing, confusing descriptions, sometimes confused with a competitor.

Skipped by AI crawlers

Without explicit robots.txt permission, cautious AI crawlers skip your site entirely. You're invisible to AI-sourced traffic.

Unreachable by agents

Agents looking for services you offer find a competitor with an MCP server instead. The conversion happens without you.

Excluded from agent marketplaces

As A2A agent networks mature, businesses without AgentCards won't appear in agent discovery. The equivalent of having no Google listing.

Questions

About implementing the five files.

Do I need all five files or can I start with just one?+

Start with llms.txt and robots.txt: those two cover Layer 1 (Discovery) and take under an hour to implement on any site. They ensure LLMs like ChatGPT and Claude can find and correctly represent you. Add mcp.json and a backing MCP server when you want agents to take actions. Add agent.json when you want to participate in agent-to-agent (A2A) networks. agents.md can be added at any stage, as it's pure documentation.

Does my site need to be built in a specific framework to support these files?+

No. All five files are static files or simple API endpoints. llms.txt, robots.txt, mcp.json, and agents.md are plain text/JSON served at fixed paths: they work on any stack including WordPress, Webflow, Squarespace, or raw HTML. The MCP server (/api/mcp) and A2A endpoint (/api/agent) require a server-side runtime, but can be deployed as standalone APIs alongside any existing site.

How quickly will these files be read by AI systems?+

GPTBot and Google-Extended typically crawl within 1–3 days of a new page being accessible. For faster indexing, submit your URLs via IndexNow (Bing's real-time indexing protocol), which also signals to Yandex and other IndexNow participants. For MCP and A2A, there's no crawl delay: agents call these endpoints in real time as needed.

What's the difference between llms.txt and agents.md?+

llms.txt is the broad overview: it tells any LLM what your business is, what you offer, and how to reach you. It's optimised for passive citation, since LLMs read it when deciding how to describe you. agents.md is the operational guide for active agents: it explains how to call your endpoints, what tasks work well, what to expect in responses, and known limitations. Think of llms.txt as your press kit and agents.md as your API documentation written in plain English.

Is this approach vendor-specific to any AI provider?+

No. llms.txt is an open format with no vendor affiliation. robots.txt is an open standard predating AI. MCP is open-sourced by Anthropic and adopted by OpenAI, Google, and others. A2A and agent.json are Linux Foundation standards backed by 150+ organisations. agents.md is a convention with no owner. The entire stack is deliberately open and vendor-neutral.

How is this different from traditional SEO?+

Traditional SEO optimises for keyword-matching algorithms that rank 10 blue links. Agentic web optimisation targets AI systems that read, reason about your content, and take actions, bypassing the link-click model entirely. An AI agent searching for a development partner won't click through a results page: it will discover your agent.json, read your agents.md, call your MCP tools, and either complete the task or escalate to a human. The conversion happens before any human sees a browser.

Get Started

Can your site survive without them?

We built all five for ourselves before we built them for anyone else. An Agentic Web audit starts from £1,500 and takes one week.

Get your audit See the full four-layer build

All five files are live on p0stman.com. Every link on this page goes to the real file.

llms.txt·robots.txt·/.well-known/agent.json·mcp.json·agents.md