p0stman
Work Services About Contact

How to Make Your Website Visible to AI

Last updated: March 2026 · 20 min read

To make your website visible to AI tools like ChatGPT, Perplexity, and Claude, you need server-side rendered content, JSON-LD structured data, discovery files like llms.txt, and direct answers at the top of every page. Sites with structured data are 2.3x more likely to appear in AI-generated answers.

Something fundamental has shifted in how people find information online. In 2025, over 130 million people used ChatGPT monthly in the US alone. Perplexity processes tens of millions of search queries. Google now shows AI Overviews on the majority of search results. The web is increasingly consumed through an AI intermediary, not a browser tab.

If your website is invisible to these AI tools, you are invisible to a growing share of your potential customers. And the gap is widening every month.

This guide covers everything you need to know about AI visibility in 2026: what determines whether AI tools reference your site, the specific technical steps to improve your visibility, and how to test whether your changes are working. It is written for business owners, marketers, and developers who want their websites to show up when AI tools answer questions relevant to their business.

Why Does AI Visibility Matter in 2026?

The business case for AI visibility is not speculative. Adobe's 2025 digital economy report found that visitors arriving from AI tools convert at 4.4x the rate of organic search visitors. These visitors also have a 45% lower bounce rate. The reason is straightforward: by the time someone clicks through from an AI recommendation, they have already been pre-qualified. The AI has matched their intent to your offering.

Consider the difference between someone searching Google for "best accounting software" and scrolling through ten blue links, versus someone asking ChatGPT "what accounting software is best for a UK freelancer who needs MTD compliance" and receiving a specific recommendation with a link to your site. The second visitor knows what they want and has been told you provide it.

The traffic numbers are also significant and growing rapidly. Gartner predicted that traditional search engine volume would drop 25% by 2026 as users shifted to AI assistants. Meanwhile, Perplexity reported growing from 10 million to over 100 million monthly queries within a single year. Microsoft reported Copilot usage exceeding 400 million monthly interactions.

For businesses, this creates a new competitive dimension. Your competitors who are visible to AI tools will capture this high-intent traffic. Those who are invisible will lose it. There is no middle ground.

AI Referral Traffic by the Numbers

Metric AI Referral Traffic Organic Search Traffic
Conversion rate 4.4x higher Baseline
Bounce rate 45% lower Baseline
Average session duration 2.3x longer Baseline
Pages per session 1.8x more Baseline
Growth rate (YoY) +900% (2024-2025) -3% to flat

The 7 Factors That Determine AI Visibility

AI tools decide whether to reference your website based on a combination of technical and content factors. Unlike traditional SEO, where Google's algorithm is a black box, the mechanics of AI visibility are more transparent. AI models reference content they can access, understand, and trust. Here are the seven factors, in order of priority.

1. Server-Side Rendering: Can AI Actually Read Your Site?

This is the single most important factor and the one most businesses get wrong. AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and others — do not execute JavaScript. When they visit your website, they receive the raw HTML that your server sends. If your website is a single-page application (SPA) that renders content via JavaScript in the browser, AI crawlers see a blank page with a few script tags.

This affects a huge number of modern websites. React applications built with Create React App, Vue applications without Nuxt, Angular applications without Angular Universal — all of these are invisible to AI crawlers by default. The crawler receives something like this: 

<!DOCTYPE html>
<html>
<head><title>My App</title></head>
<body>
  <div id="root"></div>
  <script src="/static/js/bundle.js"></script>
</body>
</html>

That empty <div id="root"> is all the AI crawler sees. Your entire site content — product descriptions, pricing, documentation, blog posts — is locked inside that JavaScript bundle and completely inaccessible.

The fix: Use a framework that supports server-side rendering (SSR) or static site generation (SSG). Next.js, Nuxt, Astro, SvelteKit, and Remix all handle this. If you are on a SPA and cannot migrate immediately, consider implementing pre-rendering for your most important pages using a service like Prerender.io or Rendertron.

How to verify: Run curl -s https://yoursite.com | head -100 in your terminal. If you can see your page content in the HTML output, you are server-rendered. If you see only script tags and an empty container, you have a problem.

2. JSON-LD Structured Data: Helping AI Understand Your Content

Structured data is metadata embedded in your HTML that tells machines what your content is about. JSON-LD (JavaScript Object Notation for Linked Data) is the format that Google, AI tools, and search engines prefer. It uses the Schema.org vocabulary to describe entities like articles, products, FAQs, organizations, and events in a way machines can parse unambiguously.

Research from multiple sources shows that pages with structured data are approximately 2.3x more likely to appear in AI Overviews and AI-generated answers. The reason is straightforward: structured data removes ambiguity. Instead of an AI model having to infer that a page is about a product with a certain price, the JSON-LD explicitly states it.

The most impactful schema types for AI visibility are:

Schema Type Use Case AI Impact
FAQPage Pages with FAQ sections Directly feeds AI question-answering
Article Blog posts, guides, news Establishes authority, freshness
HowTo Step-by-step instructions Powers procedural AI answers
Product Product pages Enables price/availability in AI responses
WebApplication SaaS tools, calculators Helps AI recommend tools
Organization Company homepage Establishes brand entity
BreadcrumbList Navigation structure Helps AI understand site hierarchy

A well-structured JSON-LD block uses the @graph pattern to combine multiple schema types on a single page. For example, a guide page should include Article, FAQPage, and BreadcrumbList in a single JSON-LD script tag. For a deeper dive into schema implementation, see our guide on JSON-LD Schema for AI.

3. Answer Capsules: Putting the Answer Where AI Looks First

AI models have a strong bias toward citing content from the first 30% of a page. Research from the AI search optimization community found that the opening section of a page is cited 44% of the time in AI-generated answers. This makes the position of your key answer critically important.

An answer capsule is a short, direct answer to the page's core question, placed at the very top of the content — before the introduction, before any preamble. It is typically one to two sentences that directly address the search query.

The implementation is simple. Wrap your direct answer in a styled container at the top of your content: 

<div class="answer-capsule">
  <p>The direct answer to the question this page answers,
  in one to two clear sentences with specific details.</p>
</div>

This serves a dual purpose. It increases the probability of being cited in AI answers, and it improves the human reading experience for visitors who arrived from an AI recommendation. They were told your page has the answer — the capsule confirms it immediately.

Every content page on your site should have an answer capsule. It does not need to be visually prominent (a subtle background and border-left is sufficient), but it must be in the DOM at the top of the article content.

4. Discovery Files: Telling AI Crawlers What Your Site Does

Traditional websites rely on robots.txt and sitemap.xml for search engine discovery. AI-visible websites add several new discovery files that specifically help AI tools understand and interact with the site.

robots.txt — Allowing AI Crawlers

The first step is ensuring AI crawlers are allowed to access your site. Many websites inadvertently block AI crawlers, either through overly restrictive robots.txt rules or by not explicitly allowing them. You should add explicit allow rules for all major AI crawlers: 

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

llms.txt — Your Site's AI-Readable Summary

llms.txt is an emerging standard that provides AI tools with a concise, structured summary of your website. Placed at your site's root (e.g., https://yoursite.com/llms.txt), it tells AI models what your site does, what content is available, what tools or APIs are exposed, and how to interact with them.

A minimal llms.txt file looks like this: 

# YourCompany

## About
YourCompany provides [what you do] for [who you serve].

## Key Pages
- /pricing - Plans and pricing
- /docs - Documentation
- /blog - Latest articles

## Contact
Email: hello@yourcompany.com
Website: https://yourcompany.com

agents.md — Extended AI Instructions

While llms.txt is concise, agents.md provides more detailed instructions for AI agents that want to interact with your site. This includes tool schemas, error handling expectations, rate limits, authentication methods, and detailed usage examples. Think of llms.txt as the summary and agents.md as the full manual.

mcp.json — Machine-Readable Tool Manifest

For sites that expose an MCP server, the mcp.json file provides a machine-readable manifest of available tools. AI agents can fetch this file to discover what capabilities your site offers programmatically.

.well-known/agent.json — A2A Agent Discovery

The Agent-to-Agent (A2A) protocol, developed by Google, uses a standardized AgentCard at /.well-known/agent.json for agent discovery. This tells other AI agents what skills your agent has and how to communicate with it.

5. MCP Servers: Letting AI Agents Use Your Tools

Discovery files make your content readable. An MCP server makes your website actionable. The Model Context Protocol (MCP) is a standard created by Anthropic that lets AI models call tools on your website — search your product catalog, check availability, retrieve pricing, book appointments, or any other interaction you expose.

With 97 million monthly SDK downloads as of early 2026 and adoption from OpenAI, Google, and Microsoft, MCP is becoming the universal standard for AI-to-tool communication. Websites with MCP servers move from being passive content sources to active participants in AI workflows.

An MCP server does not require a massive engineering effort. A basic implementation exposes 2-5 tools through a single API endpoint and can be built in under a day. The next evolution is WebMCP, which lets browsers register tools directly with AI assistants through the navigator.modelContext API (shipping in Chrome 146+).

6. Content Quality: What AI Models Consider Authoritative

AI models are trained on and retrieve content that demonstrates expertise, specificity, and trustworthiness. The same E-E-A-T principles that Google uses translate directly to AI visibility, but with some important nuances.

Depth over volume. AI models are far better at evaluating content quality than older search algorithms. A single 8,000-word authoritative guide will outperform fifty 500-word blog posts in AI citations. The model can recognize genuine expertise and tends to cite the most comprehensive source on a topic.

Specificity. AI models prefer content with specific numbers, dates, thresholds, and examples. "Landlords must give 2 months notice under Section 21" is citable. "Landlords should give appropriate notice" is not. Every claim should be specific enough that an AI can extract a factual statement from it.

Freshness. AI search tools like Perplexity and ChatGPT with browsing prioritize recent content. Include publication and update dates on every page. Keep content current with the latest data, legislation, and market conditions.

Declarative prose. Write in statements, not musings. "UK landlords must register with HMRC for Self Assessment" is AI-citable. "Many landlords wonder whether they need to register with HMRC" is not. Lead with the answer, then elaborate.

Structured content. Use clear heading hierarchies (H2, H3) where each heading is itself a question or precise sub-topic. Use comparison tables, numbered lists, and worked examples. AI models can extract information from structured content far more reliably than from flowing prose.

7. Technical SEO Foundations: The Basics Still Matter

AI visibility builds on traditional technical SEO, not replaces it. Several foundational elements remain critical:

Step-by-Step Implementation Guide

Implementing AI visibility is best done in stages, starting with the highest-impact changes that require the least effort. Here is the recommended order:

Stage 1: Foundations (Day 1)

These changes take minutes and have an outsized impact on visibility.

  1. Verify SSR. Run curl -s https://yoursite.com/ | grep -c "your-content-text". If the count is 0, your content is not server-rendered. This must be fixed before anything else matters.
  2. Update robots.txt. Add explicit allow rules for all AI crawlers listed above. This takes 5 minutes and immediately unblocks AI indexing.
  3. Add JSON-LD to your homepage. At minimum, add Organization schema with your company name, URL, description, and contact information.
  4. Submit sitemap to search consoles. Both Google Search Console and Bing Webmaster Tools. ChatGPT relies on Bing's index.

Stage 2: Content Optimization (Week 1)

  1. Add answer capsules to your top 10 pages by traffic. Identify the question each page answers and write a 1-2 sentence direct answer at the top of the content.
  2. Add FAQPage schema to every page that has an FAQ section (or add FAQ sections to your key pages). Each FAQ should have 6-10 questions with concise, factual answers.
  3. Add Article schema to all blog posts and guide pages. Include author, datePublished, dateModified, and description.
  4. Review content quality. Ensure your most important pages have at least 2,000 words of substantive content with specific, citable facts.

Stage 3: Discovery Layer (Week 2)

  1. Create llms.txt. Summarize your site, key pages, and capabilities in under 100 lines. Place at /llms.txt.
  2. Create context.md. A deeper context document covering your company, products, pricing, features, and technical details. Place at /context.md.
  3. Create an AI context API endpoint. A JSON endpoint that returns structured information about your business for programmatic AI access.

Stage 4: Action Layer (Month 1)

  1. Build an MCP server. Start with 2-3 read-only tools (e.g., search content, get pricing, list services). Deploy as an API endpoint.
  2. Create mcp.json. Manifest file listing your MCP endpoint and available tools.
  3. Add WebMCP registration. Client-side component that registers public tools with the browser for Chrome 146+.
  4. Add data-mcp-tool attributes. Annotate interactive elements (signup forms, search bars, CTAs) so browser AI agents understand what they do.

Stage 5: Agent-to-Agent (Month 2-3)

  1. Create an A2A endpoint. Implement the Agent-to-Agent protocol so other AI agents can communicate with your agent directly.
  2. Publish an AgentCard. Place at /.well-known/agent.json describing your agent's skills and capabilities.
  3. Register on agent directories. Submit your agent to directories like a2aregistry.org for discovery by other agents.

For a comprehensive view of this entire architecture, see our guide on agentic web architecture.

How to Test If Your Site Is Visible to AI

Testing AI visibility requires checking multiple pathways. Here are the four methods you should use, from simplest to most thorough.

Test 1: Direct AI Queries

The most direct test is simply asking AI tools questions that your content answers. Use ChatGPT, Perplexity, Claude, and Gemini to ask specific questions related to your business and check whether your site is cited in the response.

Be specific in your queries. Instead of "best accounting software," ask "what accounting software handles UK MTD filing for freelancers" — the more specific the question, the more likely your niche content gets cited.

Important caveat: Claude's knowledge has a cutoff date, so it can only reference content published before that date. ChatGPT with browsing and Perplexity access the live web, so they can reference newer content.

Test 2: Structured Data Validation

Use Google's Rich Results Test (search.google.com/test/rich-results) to verify your JSON-LD is valid and recognized. Also check with Schema.org's validator. Both tools will flag errors in your structured data that would prevent AI tools from parsing it correctly.

Test 3: Server Log Analysis

Monitor your server logs (or Vercel function logs, Cloudflare analytics, etc.) for visits from AI crawler user agents. Look for:

Crawler User Agent String Organization
GPTBot Mozilla/5.0 ... GPTBot/1.2 OpenAI (ChatGPT)
ClaudeBot ClaudeBot/1.0 Anthropic (Claude)
PerplexityBot PerplexityBot/1.0 Perplexity AI
Google-Extended Google-Extended Google (Gemini training)
ChatGPT-User ChatGPT-User OpenAI (live browsing)
Applebot-Extended Applebot-Extended Apple Intelligence

If you see regular visits from these crawlers, your site is being accessed by AI tools. If you see none, check your robots.txt for accidental blocks.

Test 4: Manual Crawl Simulation

Simulate what AI crawlers see by fetching your pages without JavaScript execution: 

# Fetch the raw HTML as an AI crawler would see it
curl -s https://yoursite.com/your-page | head -200

# Check for specific content
curl -s https://yoursite.com/your-page | grep -i "answer-capsule"

# Verify JSON-LD is present
curl -s https://yoursite.com/your-page | grep "application/ld+json"

# Check robots.txt
curl -s https://yoursite.com/robots.txt

If your content appears in the curl output, AI crawlers can see it. If it does not, you have a rendering or SSR issue to fix.

Common Mistakes That Block AI Visibility

After auditing dozens of websites for AI visibility, these are the mistakes we see most frequently:

Blocking AI Crawlers in robots.txt

Many websites have a blanket Disallow: / for user agents they don't recognize, or use robots.txt templates that predate AI crawlers. Some publishers deliberately block AI crawlers to protect their content — a valid choice, but one that makes the site invisible to AI tools. If your goal is AI visibility, you must explicitly allow the AI crawlers listed above.

Client-Side Rendering Without Fallback

This is the single most common technical barrier. If you're using React, Vue, or Angular without SSR, your content is invisible to every AI crawler. The fix is not optional — it is a prerequisite for all other AI visibility work.

Thin Content Pages

A 300-word page about "AI for business" will never be cited by AI tools when there are 8,000-word authoritative guides on the same topic. AI models have access to the entire web and will always prefer the most comprehensive, specific source. If you cannot write 2,000+ words of genuinely useful content on a topic, it is better not to publish the page at all.

Missing or Invalid Structured Data

Having JSON-LD that contains syntax errors is worse than having no JSON-LD at all. Invalid schema can cause AI tools to misinterpret your content. Always validate using Google's Rich Results Test before deploying.

Burying the Answer

Starting pages with "In this comprehensive guide, we will explore..." before getting to the actual answer. AI models scan for the answer in the first few hundred words. If it's buried after 500 words of preamble, the model may cite a competitor who leads with the answer instead.

Ignoring Bing

ChatGPT's search functionality is powered by Bing, not Google. If you have never submitted your sitemap to Bing Webmaster Tools, your site may not appear in ChatGPT's browsing results even if it ranks well on Google. Submit to both search engines.

No Freshness Signals

AI tools with real-time search prefer recent content. A guide published in 2022 with no update date will be deprioritized compared to one clearly marked "Last updated: March 2026." Add publication and modification dates to every page, both in the visible content and in your schema.

How Each AI Tool Discovers Your Website

Understanding how each major AI tool finds and uses web content helps you prioritize your optimization efforts. Each platform has a different discovery mechanism.

ChatGPT (OpenAI)

ChatGPT uses two discovery pathways. First, its base knowledge comes from training data with a knowledge cutoff date — content published before this date may be referenced from training. Second, ChatGPT's browsing mode uses Bing's search index to find and cite current web content in real-time.

ChatGPT also deploys the GPTBot crawler to index content specifically for training and retrieval. When ChatGPT cites your site, it automatically appends ?utm_source=chatgpt.com to outbound links, giving you attribution data.

Priority optimizations: Submit sitemap to Bing, allow GPTBot and ChatGPT-User in robots.txt, lead with direct answers.

Perplexity

Perplexity operates its own search crawler (PerplexityBot) and also uses third-party search indexes. It processes queries in real-time, searching the live web for every response. Perplexity is the most transparent about citations — every claim in its response links to a source.

Perplexity tends to favor content that is specific, recent, and well-structured. It performs particularly well with content that has clear heading hierarchies, as it can extract specific sections to answer specific questions.

Priority optimizations: Allow PerplexityBot, strong heading hierarchy, specific/quantified claims, recent publication dates.

Claude (Anthropic)

Claude's responses come primarily from its training data, which has a fixed knowledge cutoff. Claude does not browse the web in real-time (though this may change). However, ClaudeBot crawls the web for training data collection, so allowing it ensures your content is included in future training.

Where Claude excels for AI visibility is through tool use. Claude can use MCP servers to interact with your website programmatically. If a user connects your MCP server to Claude, it can search your content, retrieve data, and perform actions directly.

Priority optimizations: Allow ClaudeBot, build an MCP server, ensure content is comprehensive and authoritative.

Gemini (Google)

Gemini is deeply integrated with Google's search infrastructure. Google's AI Overviews use the same search index as traditional Google Search, which means all your existing Google SEO work directly benefits Gemini visibility. Gemini also uses the Google-Extended crawler for training data.

Gemini particularly benefits from structured data because Google's entire search infrastructure is built around schema.org understanding. Pages with valid, comprehensive JSON-LD are significantly more likely to appear in AI Overviews.

Priority optimizations: Comprehensive JSON-LD, Google Search Console optimization, allow Google-Extended, submit to IndexNow.

Apple Intelligence

Apple Intelligence, embedded in Siri and Safari, uses Applebot-Extended to crawl and index content. With iOS 18 and macOS Sequoia rolling out Apple Intelligence features, this is an increasingly important channel, particularly for consumer-facing businesses.

Priority optimizations: Allow Applebot-Extended, mobile-optimized content, clear meta descriptions, structured data.

AI Tool Discovery Matrix

AI Tool Real-Time Web Crawler MCP Support Citation Style
ChatGPT Yes (via Bing) GPTBot, ChatGPT-User Yes Inline links
Perplexity Yes (own + third-party) PerplexityBot No Numbered citations
Claude No (training data) ClaudeBot Yes (native) References in context
Gemini Yes (Google Search) Google-Extended Yes AI Overview cards
Copilot Yes (Bing) Bingbot Yes Inline links

The Four-Layer Agentic Web Framework

Everything discussed in this guide fits into a structured framework we call the agentic web. It consists of four layers, each building on the previous:

Layer 1: Discovery

How AI agents find your site. Includes robots.txt rules, llms.txt, agents.md, mcp.json, and .well-known/agent.json. This is the foundation — without discovery, nothing else matters.

Layer 2: Comprehension

How AI agents understand your content. Includes JSON-LD structured data, answer capsules, the /api/ai/context endpoint, and content quality. This determines whether your content gets cited accurately.

Layer 3: Action

How AI agents interact with your product. Includes your MCP server, WebMCP browser registration, and data-mcp-tool HTML attributes. This turns your site from a content source into an interactive tool.

Layer 4: Agent-to-Agent

How AI agents communicate with your agent. Includes the A2A protocol endpoint and AgentCard. This enables your product to participate in multi-agent workflows autonomously.

For a deep dive into each layer with implementation examples, read the full agentic web architecture guide.

Real Example: How p0stman.com Achieves AI Visibility

Rather than presenting theoretical advice, here is how we implemented AI visibility on this very website. p0stman.com is a Next.js 15 application deployed on Vercel with Supabase as the backend.

Discovery Layer

Comprehension Layer

Action Layer

Agent-to-Agent Layer

Monitoring

The result is a website that is not just visible to AI tools, but actively participates in AI workflows. AI agents can discover what we do, understand our services, use our tools, and communicate with our agent — all through standardized protocols.

AI Visibility Priority Matrix

If you can only do five things, do these in this order:

Priority Action Impact Effort
1 Ensure server-side rendering Critical (blocker) Low-High (depends on stack)
2 Allow AI crawlers in robots.txt Critical (blocker) 5 minutes
3 Add JSON-LD structured data High (2.3x improvement) 1-2 hours per page
4 Add answer capsules to key pages High (44% citation rate) 15 min per page
5 Create llms.txt Medium 30 minutes

Frequently Asked Questions

How do I get my website to show up in ChatGPT?

To appear in ChatGPT responses, your website needs server-side rendered content that ChatGPT's crawler (GPTBot) can read, JSON-LD structured data to help it understand your content, and direct answers at the top of each page. You should also allow GPTBot in your robots.txt and create an llms.txt file that summarizes your site for AI consumption. ChatGPT prioritizes content that is authoritative, specific, and well-structured.

What is llms.txt and do I need one?

llms.txt is a plain text file placed at your website's root (like robots.txt) that provides a concise, machine-readable summary of your site for AI models. It typically includes what your site does, key pages, available tools or APIs, and quick start instructions. While not required, having an llms.txt file significantly improves how AI tools understand and reference your content.

Can AI tools read JavaScript-rendered websites?

Most AI crawlers cannot execute JavaScript. If your website is a single-page application (SPA) built with React, Vue, or Angular that renders content client-side, AI tools will see an empty page. You need server-side rendering (SSR) or static site generation (SSG) to ensure AI crawlers can read your content. Next.js, Nuxt, and Astro all support this out of the box.

How long does it take for AI tools to index my website?

There is no guaranteed timeline. ChatGPT's search index (powered by Bing) can pick up new content within days to weeks. Perplexity indexes content in near real-time through its search crawlers. Claude's training data has a knowledge cutoff, so it only knows about content published before that date. The best strategy is to optimize for all pathways: search indexing, real-time crawling, and structured data that AI tools can access programmatically.

What is the difference between SEO and AI visibility?

Traditional SEO optimizes for search engine rankings and click-through rates. AI visibility goes further: it optimizes for how AI models understand, cite, and interact with your content. This includes structured data for machine comprehension, discovery files like llms.txt for AI-specific crawlers, MCP servers that let AI agents use your tools directly, and answer capsules that increase citation rates. AI visibility builds on SEO but adds a programmatic interaction layer.

Do I need an MCP server for AI visibility?

An MCP server is not required for basic AI visibility but is important for advanced AI interaction. If you just want your content to appear in AI responses, structured data and good content are sufficient. If you want AI agents to actually use your product — search your catalog, book appointments, retrieve data — you need an MCP server. Think of it as the difference between being readable by AI and being actionable by AI.

Which AI crawlers should I allow in robots.txt?

You should allow all major AI crawlers: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Gemini), PerplexityBot, Applebot-Extended (Apple Intelligence), cohere-ai, Meta-ExternalAgent, Amazonbot, Bytespider, Diffbot, YouBot, CCBot, and OAI-SearchBot. Blocking these crawlers means your content cannot appear in AI-generated responses from those platforms.

How do I test if my website is visible to AI?

Test AI visibility in four ways: (1) Ask ChatGPT, Perplexity, and Claude direct questions that your content answers and see if you are cited. (2) Use Google's Rich Results Test to verify your structured data is valid. (3) Check your server logs for AI crawler visits from GPTBot, ClaudeBot, and PerplexityBot. (4) Use tools like Screaming Frog or Ahrefs to verify your pages are server-rendered and crawlable. For programmatic testing, fetch your pages with a simple curl command to confirm the HTML contains your content without JavaScript execution.

Make Your Website AI-Ready

We help businesses implement AI visibility across all four layers of the agentic web — from structured data and discovery files to MCP servers and A2A endpoints. Get an AI readiness audit for your website.

Agentic Web Readiness Talk to Us

Continue Reading

What Is an MCP Server?

Model Context Protocol explained — let AI agents use your tools

What Is llms.txt?

The discovery file that tells AI tools what your site does

Agentic Web Architecture

The four-layer framework for AI-native websites