Agentic Web Architecture

Before an AI agent can understand your content or use your tools, it needs to know you exist. Discovery is the foundation layer — the collection of files and signals that tell AI agents "this website is here, and here's what it does."

Traditional websites relied on Google indexing to be found. AI agents use a different set of discovery mechanisms. Some overlap with SEO (robots.txt, sitemaps), but many are entirely new protocols that emerged specifically for the agentic web.

The first gate is the oldest file on the web: robots.txt. Many websites inadvertently block AI crawlers because their default robots.txt only permits Googlebot and Bingbot. In 2026, there are at least 15 AI-specific crawlers that check your robots.txt before indexing your content.

If your robots.txt blocks these crawlers — or worse, uses a blanket Disallow: / for unknown user agents — your content is invisible to every AI model except those that ignore robots.txt (which is a declining minority as the industry self-regulates).

Here is the complete list of AI crawlers you should explicitly allow:

Key implementation detail: use explicit Allow: / rules rather than relying on the absence of Disallow. Some AI crawlers treat missing rules as "proceed with caution" rather than "allowed." Explicit permission is clearer.

Also allow API paths that agents might call directly:

llms.txt is a plain-text file at the root of your website that provides a concise summary of what your site does, what tools are available, and how to interact with it. Think of it as a README for AI models.

The standard was proposed by Jeremy Howard (fast.ai) in late 2024 and has seen roughly 10% adoption across commercial websites as of early 2026. Its adoption is growing because it solves a real problem: when an AI model lands on your site, it needs to quickly understand what it's looking at without parsing hundreds of pages of HTML.

The file sits at public/llms.txt and should contain:

• What the site/product does (1-2 paragraphs)
• Key features or capabilities
• Available tools/APIs (if any)
• Authentication method
• Quick-start curl examples
• Links to deeper documentation

Keep it under 100 lines. AI models have context windows, and a concise llms.txt that fits entirely in one context window is more useful than a sprawling document the model needs to summarise.

Where llms.txt is a summary, agents.md is the full manual. This Markdown file at public/agents.md provides detailed instructions that AI agents use when interacting with your site.

While llms.txt tells an agent "here's what we do," agents.md tells it "here's exactly how to work with us." It should include:

• Detailed tool schemas with parameter descriptions
• Error handling instructions (what to do when a tool returns an error)
• Rate limits and usage policies
• Authentication flows (step-by-step)
• Example workflows for common tasks
• Data formats for requests and responses

This file becomes essential once you have Layer 3 (Action) tools. Without it, AI agents will attempt to use your tools through trial and error, which leads to poor user experiences and wasted API calls.

mcp.json is a JSON file at your site root that provides a structured, machine-parseable manifest of your MCP capabilities. While llms.txt and agents.md are for AI models to read as context, mcp.json is for AI agent frameworks to parse programmatically.

The critical fields are endpoint (where to send MCP requests), auth (how to authenticate), and tools (what's available). Agent frameworks use this to auto-configure tool access without human intervention.

The AgentCard is the A2A protocol's discovery mechanism. Located at .well-known/agent.json, it tells other AI agents about your agent's capabilities, skills, and communication endpoint.

This is the Layer 4 discovery file — it's how your agent announces itself to other agents. Without it, no A2A communication is possible because other agents have no way to discover your agent's skills or endpoint.

Discovery tells AI agents you exist. Comprehension tells them what you actually do, what you know, and whether your content is trustworthy. This is the layer that determines whether AI models cite you, recommend you, or ignore you.

Comprehension has the highest return on investment of any layer. The techniques here — JSON-LD structured data, answer capsules, and context endpoints — directly influence whether your content appears in AI Overviews, ChatGPT responses, and Perplexity citations.

JSON-LD (JavaScript Object Notation for Linked Data) is structured metadata embedded in your page's <head> section. It tells machines — search engines and AI models alike — the precise meaning of your content using a standardized vocabulary (Schema.org).

The impact is substantial: pages with proper JSON-LD structured data are 2.3x more likely to appear in AI Overviews (Authoritas study, 2025). AI models use structured data not just for ranking but for fact extraction — when a model needs to answer "how much does X cost?" it looks for Product schema with price properties before parsing free-text content.

The most impactful schema types for AI visibility, in order:

Use the @graph pattern to combine multiple schema types on a single page. A guide page should include Article + FAQPage + BreadcrumbList in a single JSON-LD block. For a deep technical reference on implementation, see our dedicated guide on JSON-LD schema for AI visibility.

AI language models cite the first 30% of page content 44% of the time. This statistic — from research by Authoritas and confirmed by multiple SEO studies in 2025 — means that where you place your answer matters enormously.

An answer capsule is a 1-2 sentence direct answer to the page's core question, placed at the very top of the content, wrapped in a <div class="answer-capsule">. It serves three purposes:

1. AI citation — Models pull from it when generating answers
2. Featured snippets — Google uses it for position zero results
3. Human UX — Visitors get their answer immediately (especially AI-referred visitors who already have context)

The answer capsule at the top of this page is an example. It directly answers "what is agentic web architecture" in two sentences, before the introduction expands on it. Every content page in your agentic web architecture should follow this pattern.

While JSON-LD is embedded in individual pages, the /api/ai/context endpoint provides a single, centralised, machine-readable summary of your entire business. AI agents call this endpoint to get a complete picture without crawling every page.

The response should be a JSON object containing:

This endpoint is public, requires no authentication, and should return a stable JSON structure. AI agents can cache it and use it as context when generating responses about your business. It's the Comprehension layer's equivalent of llms.txt, but structured for machine parsing rather than human reading.

Beyond schema and answer capsules, how you structure your content affects how well AI models can extract and cite it. The principles:

• Every H2/H3 should be a standalone answerable question. "How much does an MCP server cost to build?" is better than "Costs." AI models use headings to segment content and match questions to answers.
• Use tables for comparative data. AI models extract table data more reliably than prose. If you're comparing features, pricing tiers, or options, use a table.
• Include specific numbers and dates. "Costs £3,000-£8,000 and takes 1-2 weeks" is citable. "Costs vary and timelines depend on scope" is not.
• Write declaratively. "Businesses implementing MCP servers see 15-40% increases in AI-referred traffic" is better than "You might see improvements in your traffic from AI sources."
• Minimum 4,000 words for substantive pages. Thin content (under 2,000 words) is consistently outperformed in AI citations by comprehensive, authoritative pages.

Layers 1 and 2 make your website readable by AI. Layer 3 makes it usable. This is where the web shifts from a collection of documents to a collection of capabilities — and where the real business value of agentic web architecture begins.

Layer 3 is built on the Model Context Protocol (MCP), an open standard that lets AI models call tools exposed by your website. When a user asks ChatGPT "book a consultation with p0stman," the AI can call your MCP endpoint directly instead of navigating your website like a human would.

An MCP server is an API endpoint — typically at /api/mcp — that exposes your website's capabilities as callable tools. The endpoint responds to two types of requests:

• GET — Returns server information: name, description, available tools, auth requirements. This is the discovery payload that AI agents read to understand what your server can do.
• POST — Executes a tool or accesses a resource. The request body specifies which tool to call and with what parameters. Authentication is typically via Bearer token.

The tools you expose depend on your business. Here are examples across different business types:

A basic MCP server implementation in Next.js:

WebMCP is the client-side complement to your server-side MCP endpoint. It uses the navigator.modelContext API (previewed in Chrome 146) to register tools directly with AI models running in the browser.

The key distinction: server-side MCP is for AI agents making API calls. WebMCP is for AI assistants embedded in the browser (like Gemini in Chrome, or ChatGPT's browsing feature) that are already on your page and want to understand what actions are available.

What to register via WebMCP:

• Lightweight, public, no-auth tools only. Things like "get_services" or "search_content" — never tools that require API keys or modify data.
• 2-3 tools maximum. Browser AI models have limited context for tool registration. Register your most useful public capabilities.
• Graceful degradation. WebMCP is Chrome-only as of March 2026. Your component must be a no-op on unsupported browsers.

The declarative approach to WebMCP. Instead of registering tools via JavaScript, you annotate HTML elements with data-mcp-tool and data-mcp-description attributes. Browser AI models scan the DOM for these attributes to understand what page elements do.

This is the lowest-effort way to make your website's interactive elements understandable to AI. Even without a full MCP server, adding these attributes to your forms, CTAs, and key navigation elements improves how browser AI models interact with your site.

Exposing tools to AI agents introduces security surface area that doesn't exist on traditional websites. Key principles:

• Separate public and authenticated tools. Read-only tools (get_services, search_content) can be public. Write tools (create_account, submit_form) must require authentication.
• Rate limit aggressively. AI agents can call tools much faster than humans click buttons. Implement per-IP and per-token rate limits on your MCP endpoint.
• Validate all inputs. AI agents may pass unexpected parameter types or values. Validate and sanitize every input, just as you would for a public API.
• Never expose secrets in WebMCP tools. Client-side tool handlers run in the browser. Never include API keys, database credentials, or internal URLs in WebMCP-registered tools.
• Log everything. Track which tools are called, by whom, and how often. This data is essential for understanding how AI agents use your site and detecting abuse.

Layer 4 is the most forward-looking part of the framework. While Layers 1-3 optimise your website for AI models that serve human users, Layer 4 enables direct agent-to-agent communication — where an AI agent representing one business talks to an AI agent representing your business, without a human in the loop.

The protocol that enables this is A2A (Agent-to-Agent), developed by Google and adopted by a growing ecosystem of agent frameworks. It uses JSON-RPC 2.0 over HTTPS, making it straightforward to implement on any web server.

The A2A endpoint handles three types of HTTP requests:

• GET — Returns the AgentCard (the same content as .well-known/agent.json), enabling discovery via the endpoint URL itself.
• OPTIONS — CORS preflight. Essential because A2A requests come from different origins.
• POST — Receives a task from another agent, processes it, and returns a response.

The A2A request/response format follows JSON-RPC 2.0:

A2A tasks follow a state machine with four possible states:

1. submitted — The requesting agent has sent a task. Your agent acknowledges receipt.
2. working — Your agent is processing the request. For streaming implementations, you can send partial updates during this state.
3. completed — Your agent has finished processing and returns the result as one or more artifacts.
4. failed — Processing failed. Return an error message explaining why.

For most implementations, the simple synchronous pattern (receive task, process, return completed) is sufficient. Streaming and push notifications are available for long-running tasks but add complexity that most businesses don't need initially.

Every A2A interaction should be logged to a database table. This serves two purposes: understanding how other agents interact with your business, and debugging when things go wrong.

Review these logs regularly. They tell you what other AI agents are asking your agent, which skills are most used, and where your agent's responses could be improved.

You do not need to implement all four layers at once. In fact, trying to build everything simultaneously is a mistake — each layer should be validated before moving to the next. Here is the recommended implementation sequence:

This framework is not theoretical — it's the architecture running on p0stman.com right now. Here is a walkthrough of every component:

The total implementation took approximately two weeks of focused development, starting from a standard Next.js site that had none of these capabilities. The bulk of the time was spent on Layer 3 (the MCP server) and Layer 4 (the A2A endpoint). Layers 1 and 2 were completed in the first day.

Since going live, the bot tracking data has shown consistent crawling from GPTBot, ClaudeBot, and PerplexityBot across all content pages, with MCP discovery requests appearing within the first week of deployment.

Traditional web analytics (Google Analytics, Plausible, Vercel Analytics) track human visitors. Agentic web architecture requires a second analytics layer that tracks AI interactions separately.

Bot Crawl Tracking

The most basic monitoring: which AI crawlers visit your site, how often, and which pages they request. Implement this in your middleware or as a lightweight API endpoint.

Detect bots by matching user-agent strings in your middleware. Here are the patterns to match:

MCP Call Logging

Log every MCP tool call to understand how AI agents use your tools. Track the tool name, input parameters, response time, and whether the call succeeded or failed. This data tells you which tools are most valuable and where to invest development effort.

A2A Session Analytics

The agent_sessions table captures every A2A interaction. Review it weekly to understand what other agents ask, which skills they use, and what questions your agent struggles to answer. This data directly informs improvements to your agent's knowledge base and skills.

LLM Referral Tracking

When AI tools like ChatGPT cite your content, they append ?utm_source=chatgpt.com (or similar) to the link. Track these UTM parameters to measure how much traffic AI citations drive. Our analytics show LLM-referred visitors have a 45% lower bounce rate and 4.4x higher conversion rate than organic search visitors.

The most important insight from this table: Layers 1 and 2 cost almost nothing but deliver the highest immediate impact. A senior developer can implement both layers in a single day. Layers 3 and 4 require more engineering but provide competitive differentiation that will become increasingly valuable as AI agent traffic grows.

The infrastructure costs are negligible. Discovery files are static (free to serve). Comprehension endpoints add minimal server load. MCP and A2A endpoints process a fraction of the traffic that human-facing pages receive. The primary cost is developer time, not infrastructure.

Having implemented this framework across multiple production sites, here are the mistakes we see most often:

1. Building Layer 3 Before Layer 2

Teams get excited about MCP servers and skip content optimization. An MCP server with great tools is useless if AI models don't know your site exists or understand what you do. Always establish Comprehension before building Action capabilities.

2. Exposing Too Many Tools

Some implementations expose 20+ MCP tools. AI models have limited context for tool selection — presenting too many tools degrades the quality of tool calls. Start with 2-3 core tools and expand based on usage data. Quality of tool descriptions matters more than quantity of tools.

3. Forgetting Authentication on Write Operations

We've seen MCP servers where tools like "create_order" or "submit_form" require no authentication. Any AI agent (or malicious script) can call these endpoints. Every tool that modifies data must require authentication.

4. No Monitoring

Building the infrastructure without tracking how it's used. You need bot crawl logs, MCP call logs, and A2A session logs to understand what's working. Without data, you're guessing.

5. Static llms.txt That Gets Stale

llms.txt should be updated whenever your site's capabilities change — new tools, new content sections, updated pricing. Treat it like a README that you maintain alongside your codebase.

6. Ignoring Answer Capsules

The single highest-impact change for AI citation is adding a direct answer at the top of every content page. It costs nothing, takes minutes per page, and directly affects whether AI models cite you. Yet it's the most commonly skipped step.

7. Missing CORS on A2A Endpoints

A2A requests come from different origins. If your /api/agent endpoint doesn't handle OPTIONS preflight requests and set appropriate CORS headers, other agents' requests will be blocked by the browser's same-origin policy.

As of March 2026, we're in the early adopter phase of the agentic web. Here's what's coming:

WebMCP Standardization (2026-2027)

Chrome 146's navigator.modelContext API is the first browser-native implementation of WebMCP. Firefox and Safari will follow with their own implementations. By late 2027, expect a W3C working group proposal for standardizing browser-to-AI-model communication. This will make WebMCP as standard as Service Workers or the Web Payments API.

MCP Registry Consolidation

Currently, MCP servers are listed across multiple directories (Smithery, mcp.so, PulseMCP). Expect consolidation into 1-2 dominant registries by 2027, similar to how npm won the JavaScript package registry war. Registering your MCP server early establishes presence.

A2A Maturity

The A2A protocol is currently at version 1.0, with limited adoption outside Google's ecosystem. As more businesses deploy AI agents that need to coordinate (travel booking, financial services, healthcare), A2A will become the standard for agent interoperability. Early implementations position you ahead of competitors who will scramble to add A2A support later.

Agentic Commerce

The most transformative shift: AI agents making purchasing decisions on behalf of users. "Find me the best web development agency for an AI project under £10,000" triggers an agent that evaluates multiple businesses by reading their llms.txt, calling their MCP tools, and comparing structured data. The businesses with full agentic web architecture will be evaluated. Those without it won't even be in the running.

AI-Native Websites

Eventually, some businesses will build websites that are agent-first and human-second. The primary interface will be tools and APIs for AI agents, with a human-readable UI as a secondary presentation layer. This sounds radical today, but it mirrors how many SaaS products already prioritise their API over their web UI for power users.