The five files every site needs.
By 2027, every agent-ready site will expose five machine-readable files: llms.txt, robots.txt, an AgentCard, an MCP manifest, and agents.md. All five let AI agents discover, understand, and act, without a human touching a browser.
- Live on p0stman.com now
- Five files, all live
- From £1,500
- Scroll
Thirty years of web design assumed a human was doing the clicking. That assumption is already breaking down.
AI agents (Claude, ChatGPT, Gemini, LangGraph orchestrators, AutoGPT successors) are now among the fastest-growing categories of web traffic. They don't use browsers. They don't click links. They read structured data, call APIs, and take actions on behalf of users.
A website that doesn't expose the right files to these agents is invisible to them. Not penalised. Just absent. The agent finds a competitor with the infrastructure instead, and the human never knows your site existed.
These are the five files that separate agent-ready websites from the rest. All five are live on p0stman.com: every link on this page goes to the real file.
What each file does, and how to build it.
llms.txt
The AI index file · Layer 1 · Discovery
A plain-text file at the root of your domain that tells large language models who you are, what you do, and where to find key information about your business.
When ChatGPT, Claude, or Perplexity decides how to describe your business to a user, it needs somewhere to start. Without llms.txt, it guesses from scraped HTML: getting your pricing wrong, misrepresenting your services, or confusing you with a competitor.
llms.txt gives you direct input into that process. It's the equivalent of a press kit, written for machines. Proposed by Jeremy Howard (fast.ai) in 2024, it's now adopted by hundreds of companies and actively read by LLM crawlers.
Examplellms.txt# p0stman > AI-native product studio. We build voice agents, AI-powered > applications, and custom software for businesses that want to > move faster than traditional agencies allow. Contact: hello@p0stman.com | https://p0stman.com MCP server: https://p0stman.com/api/mcp ## Services ### AI Voice Agents: from £5,000 | 6-10 days Inbound and outbound voice AI for hotels and restaurants. ### MVP Launch: from £5,000 | 1-3 weeks From idea to live product in weeks.How to implementLive example: p0stman.com/llms.txt (opens in a new tab)Create a file named llms.txt at the root of your site. Write in plain markdown. Include: company description, services with pricing, contact details, and key page URLs. 500–2,000 words. No HTML. Serve it at the root.
robots.txt
The AI permissions file · Layer 1 · Discovery
The standard file that tells web crawlers what they can and cannot access, updated to explicitly allow the new generation of AI crawlers that most sites were not built to accommodate.
Most robots.txt files were written before AI crawlers existed. A blanket Disallow or an unconfigured file means AI systems from OpenAI, Anthropic, and Google may skip your site entirely, or only partially index it.
The new AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) check robots.txt before reading anything. If they're not explicitly allowed, the safest implementations will skip you. Worse, some will allow crawling but not attribute your content correctly without explicit permission signals.
Examplerobots.txtUser-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Googlebot Allow: / Disallow: /admin Disallow: /api/admin Sitemap: https://yoursite.com/sitemap.xmlHow to implementLive example: p0stman.com/robots.txt (opens in a new tab)Open your existing robots.txt and add explicit Allow rules for each AI crawler. The key bots to allow: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, TavilyBot. Keep your existing rules for admin and private areas.
/.well-known/agent.json
The AgentCard · Layer 4 · Agent-to-Agent
A machine-readable JSON file served at the RFC 8615 well-known path that declares your AI agent's identity, capabilities, and task endpoint: the business card of the agentic web.
The A2A (Agent-to-Agent) protocol, backed by Google, Microsoft, IBM, and 150+ organisations under the Linux Foundation, defines how AI agents discover and communicate with each other. The AgentCard is the discovery mechanism.
When an AI agent (say, a customer's personal AI assistant) wants to interact with your business autonomously, it checks /.well-known/agent.json first. Without it, your business doesn't exist to that agent. With it, the agent knows your name, what you can do, and exactly how to send you a task.
The well-known path (RFC 8615) is intentional: the same convention used by ACME challenges, security.txt, and OpenID Connect. Every A2A-compatible system knows to look there.
Example/.well-known/agent.json{ "name": "Zero", "description": "AI ops assistant at p0stman", "version": "1.0", "url": "https://p0stman.com/api/agent", "capabilities": { "streaming": false, "pushNotifications": false }, "skills": [ { "id": "inquire", "name": "Agency Inquiry", "description": "Answer questions about services and pricing" }, { "id": "book", "name": "Book Discovery Call", "description": "Schedule a free 30-minute discovery call" } ], "authentication": { "schemes": ["None"] }, "provider": { "organization": "p0stman", "url": "https://p0stman.com" } }How to implementLive example: p0stman.com/.well-known/agent.json (opens in a new tab)Create a JSON file at public/.well-known/agent.json (or serve it as a route at /.well-known/agent.json). Include: name, description, version, the URL of your A2A task endpoint, capabilities, skills with IDs and descriptions, and authentication scheme. Pair it with an actual /api/agent endpoint that accepts JSON-RPC tasks/send requests.
mcp.json
The MCP tool registry · Layer 3 · Action
A JSON manifest listing the tools (actions) your site exposes via the Model Context Protocol: the file that tells AI assistants what they can actually do on your site.
MCP (Model Context Protocol), open-sourced by Anthropic and now adopted by OpenAI, Google, and dozens of AI platforms, defines how AI agents call tools on external services. Claude Desktop, Cursor, and a growing list of AI assistants can read mcp.json and offer your tools directly to their users.
Without mcp.json and a backing MCP server, your business is read-only to AI agents. They can learn about you from llms.txt, but they can't act. With MCP, an agent can check your availability, submit an enquiry, or book a call on behalf of a user, all within their AI interface, without visiting your site.
This is where passive discovery becomes active integration.
Examplemcp.json{ "name": "p0stman AI Studio", "description": "AI-native product studio. Book calls, browse services, view portfolio.", "version": "1.0.0", "endpoint": "https://p0stman.com/api/mcp", "protocol": "json-rpc-2.0", "tools": [ { "name": "book_discovery_call", "description": "Book a free 30-minute discovery call." }, { "name": "submit_inquiry", "description": "Submit a project inquiry." }, { "name": "get_services", "description": "Get services with pricing and timelines." } ] }How to implementLive example: p0stman.com/mcp.json (opens in a new tab)Create mcp.json at your site root listing your tools with names and descriptions. Build a corresponding MCP server at /api/mcp that handles tools/list (returns all tools with inputSchema) and tools/call (executes the tool and returns structured results). Use JSON-RPC 2.0 format.
agents.md
The capability manifest · Layer 1 · Discovery
A markdown file that bridges the gap between the brief llms.txt and the structured agent.json: giving AI agents detailed, nuanced context about what your agent can do and how to work with it effectively.
llms.txt tells LLMs who you are. agent.json tells other agents how to call you. agents.md tells agents what to actually do and what to expect: the nuance that structured JSON can't capture.
It's where you explain edge cases, preferred task formats, example queries that work well, limitations to be aware of, and capabilities not covered by the formal schema. Think of it as the README for your agent.
As multi-agent orchestration matures, agents will read agents.md before deciding whether to delegate a task to your agent, how to phrase the request, and what format to expect back. The more useful and specific it is, the more reliably other agents can work with yours.
Exampleagents.md# p0stman: Agent Instructions ## About AI-native product studio. Builds voice agents, MVPs, web apps for businesses. ## MCP Server (Recommended) Endpoint: https://p0stman.com/api/mcp Protocol: JSON-RPC 2.0 ## A2A Task Endpoint Endpoint: https://p0stman.com/api/agent Protocol: JSON-RPC 2.0 (tasks/send method) ## Example Tasks - "What AI services do you offer for restaurants?" - "What would an AI voice agent cost for a 3-site hotel?" - "Book a discovery call for [name] at [email]" ## Limitations - Cannot process payments directly - Discovery calls must be confirmed by emailHow to implementLive example: p0stman.com/agents.md (opens in a new tab)Create agents.md at your site root. Include: who you are (1 paragraph), MCP endpoint with example calls, A2A endpoint with example tasks, what requests work well, known limitations, and contact fallback. Write for a machine reader: structured, specific, no marketing fluff.
How the five files work together.
What happens to sites that don't have them.
Not penalised. Just absent, in the exact moment a buying decision is being made.
About implementing the five files.
Do I need all five files or can I start with just one?
Does my site need to be built in a specific framework to support these files?
How quickly will these files be read by AI systems?
What's the difference between llms.txt and agents.md?
Is this approach vendor-specific to any AI provider?
How is this different from traditional SEO?
All five files are live on p0stman.com. Every link on this page goes to the real file.
We built all five for ourselves before we built them for anyone else. An Agentic Web audit starts from £1,500 and takes one week.
