How to Appear in AI Search Results:
3 Steps That Actually Work

How to appear in AI search results: the quick answer

To appear in AI search results, your website needs three things. AI crawlers must be able to access your pages (check robots.txt, your CDN, and your firewall). Your identity must be machine-readable, which means publishing AI Discovery Files such as llms.txt and identity.json alongside Schema.org markup. And your business details must be consistent everywhere AI systems verify them, from your own site to Google Business Profile and LinkedIn.

If you would rather know where you stand before changing anything, two checks need no work at all. Can AI find your website today? Your Search Console generative AI report says whether AI systems are already using your pages, and the Search generative AI control in Settings says whether something has switched you off. Both are covered in the section on reading your own Search Console.

We've watched these steps work from a standing start. A hand-built locksmith site launched in April 2026 with zero backlinks was the #1 answer on ChatGPT and Gemini for its head term within twelve days. The full case study documents exactly what was deployed. The rest of this guide explains why each step matters and how to carry it out on your own site.

Why AI search visibility matters now

ChatGPT now has over 900 million weekly active users. Google AI Overviews appear in roughly half of all search queries. Perplexity processes hundreds of millions of searches per month. AI isn't coming for traditional search. It's already absorbing it. For a breakdown of how these tools compare as consumer subscriptions, see which AI subscription is best for the average user.

For website owners, this creates a new problem. Your site might rank well on Google and still be completely invisible to the AI systems that are answering an increasing share of user questions. When someone asks ChatGPT "Who provides web design in Kettering?" or asks Gemini "What's the best CRM for small businesses?", the answer isn't a list of links. It's a direct, synthesised response that either includes your business or doesn't. We covered the small-business angle in detail in AI Visibility for Small Businesses: What It Actually Delivers, which uses the Lockerfella case as the worked example.

The numbers back this up. AI referral traffic grew 357% year-over-year in 2025. That traffic converts at five times the rate of traditional organic search. Visitors arriving from AI platforms spend 68% more time on site. The traffic is smaller than Google's, but it's higher quality and growing fast.

This guide covers what AI search engines actually need from your website, why most sites fail to provide it, and what you can do about it today.

How AI search engines find your website

AI search engines don't work like Google. Understanding the difference is the first step to appearing in their results.

Traditional search engines crawl your site, index its pages, and rank them against competing pages for specific queries. The output is a list of links. You compete for position.

AI search engines use a process called retrieval-augmented generation (RAG). When a user asks a question, the AI system searches for relevant web pages in real time, retrieves and reads them, then generates a synthesised answer. The output isn't a list. It's a paragraph (or several) with citations linking back to the sources used. You don't compete for position. You compete for inclusion. We break this down stage-by-stage in the full retrieval pipeline explainer.

ChatGPT triggers a web search on an estimated 20-35% of its prompts, which works out to 500-875 million web-retrieval queries per day. That's about one in eleven of Google's daily search volume, and it's growing.

What matters in this model isn't keyword density or backlink profiles. What matters is whether your content is accessible, clearly structured, and machine-readable. The AI needs to be able to crawl your page, understand what your business does, and trust the information enough to cite it.

Three things determine whether your website makes the cut: access, clarity, and trust. The three steps in this guide map directly onto them.

Why your website isn't showing up in AI search results

Here is the finding that surprises most people who ask us why they can't get found in AI search: you are probably not blocked. You are silent.

Every quarter our ADF adoption crawler fetches robots.txt and the ten AI Discovery File paths from roughly two thousand of the most visited websites in the world and the UK. The Q3 2026 crawl completed on 1 July 2026 and successfully read 1,744 sites. It scores each one into a readiness tier. The distribution is lopsided.

75.5% of sites sit in the Passive tier. AI crawlers can reach them perfectly well. There is simply nothing on the site that tells a machine who the business is, what it sells, or where it operates, beyond prose written for humans. Only 1.3% block every AI crawler outright. Deliberate blocking is real, and we cover it in the blocking checklist, but it is nowhere near the main reason businesses can't get found in AI search.

The supporting numbers point the same way. Across those 1,744 sites, 90.6% published no AI Discovery File of any kind, 73.7% had no Schema.org structured data on the homepage, and 84.2% had no AI crawler policy in robots.txt at all: no rule blocking them, no rule allowing them, nothing. Not one site in the sample reached the top AI-Optimised tier.

If your website isn't showing up in AI search results, work through these six causes in order. Each one has to be ruled out before the next one is worth investigating.

There is one further possibility worth naming honestly, because no amount of technical work removes it: the model may simply have picked someone else this time. Retrieval is not deterministic, and the same prompt run twice can return different sources. That is a genuine limit on what any of this can promise, and we come back to it in the myths section. What the six causes above give you is control over the part that is deterministic: whether you were eligible to be picked at all.

Are you already showing up in AI search? Read your own Search Console

You may already be showing up in AI search without knowing it. Impressions from Google's AI surfaces have always been folded into your ordinary Search Console totals, so a site being used as a source inside AI Mode looks, at a glance, like a site having a good month. Since 31 August 2026 you can separate the two, and it costs nothing.

That matters because it is the honest answer to a question we are asked constantly and used to answer badly: how do I know whether AI can find my website? Until this summer the only first-party evidence available to a site owner was their own server logs. Now Google reports it.

The report Google added, and the number it will not give you

Google launched a dedicated Generative AI performance report in Search Console on 3 June 2026, first to a limited set of properties, then to every website worldwide on 31 August 2026. It covers AI Overviews, AI Mode, and generative AI features in Discover. Data starts on 18 May 2026 with no backfill, so nobody has a longer history than that.

It reports impressions, broken down by page, country, device and date. It does not report clicks, click-through rate, or average position. Google's documentation is explicit that this is not a separate pot of traffic: the report includes data from the Web search type in the Performance report (Search results). Your AI impressions were already in your totals. You simply could not see which ones they were.

What AI Mode impressions look like in your Search Console data

Here is this page in Google Search Console for the three months to 2 September 2026: 27,756 impressions, 152 clicks, average position 9.04. Read as one number, that is a page with a click-through-rate problem. Split the queries by length and it stops reading that way at all.

All 203 long queries produced zero clicks between them, at an average position of 5.4. Not a poor click-through rate. Zero, across 14,661 impressions, 13,903 of which sat inside the top ten. Every attributable click on the page came from the short keyword queries, where we rank around 26th.

Two queries account for 65% of the named impressions, and neither is a keyword:

Nobody types those. They are the shape of a question asked in a chat box. AI Mode answers a question by issuing several related searches at once, a technique Google calls query fan-out, so one person asking one thing generates a spread of machine-written queries, each able to register an impression against whichever pages get retrieved. That is why the wording is so conversational, why one of them turns up with a stray is it bolted on the front, and why 92% of the impressions came from desktop.

So the signature of being used as an AI source, visible in your own Search Console, is this: long question-shaped queries, a top-ten average position, and a click-through rate at or near zero. If that is your pattern, you are not failing to earn clicks. You are being read.

Where we got this wrong, and what three months proved

I want to be straight about this, because we made the mistake ourselves and the correction is the useful part.

In June 2026 this page showed impressions climbing and clicks flat, so we diagnosed a snippet problem. We rewrote the meta title to match the query family, front-loaded the description with proof, added a quick-answer block as a featured-snippet target, and shipped it on 10 June. The work was competent and the reasoning was sound on the evidence we had. It was also aimed at the wrong thing.

Three months of data settle it. Impressions went from 4,257 in the first month to 15,107 in the third, a rise of 3.5x. Clicks went from 40 to 68. Click-through rate fell from 0.94% to 0.45%, and it fell because of that growth rather than in spite of it, since almost all the new impressions arrived on a surface where clicking is not what happens next. No snippet rewrite was going to move that number, because the number was counting an event the surface does not produce.

The lesson is not that click-through rate stopped mattering. It is that a falling CTR on a page whose impressions are tripling can mean the opposite of what it looks like, and you cannot tell which without splitting the queries. Anyone reporting AI-era search performance to a client or a board needs that split, or they will present a success as a failure.

One caveat, stated plainly. The main Performance report does not label which impressions came from an AI surface, and the Generative AI report gives a total without breaking it down by query. The long-query pattern above is therefore a signature rather than a label. What the Generative AI report does give you is a headline number to check that signature against, which is more than anyone had in May.

The Search Console switch that can turn your AI visibility off

Before you change anything on your site, check one setting. In Search Console, under Settings, there is now a Search generative AI control with three states: include your site, exclude your site, or inherit the value from a parent property. It rolled out worldwide on the same day as the report.

Include is the default for every property, so most sites are fine. If it has been set to exclude, Google is unambiguous about what follows: links to your site and your site's content won't appear in Search generative AI features. You won't receive any traffic or impressions from these features. A flawless robots.txt, a complete set of discovery files and perfect schema will not get you back in. Changes are respected on a rolling basis, generally within one to two days, though caching can stretch that out.

This is a real change in what site owners control, and it cuts both ways. Until now, leaving AI Overviews meant blocking Googlebot, which also removed you from Google Search: an all-or-nothing trade almost nobody should take. There is now a switch for it, and Google says the choice is not used as a ranking signal in ordinary web search. If you have a genuine reason to stay out of AI answers, that is the clean way to do it, and our blocking guide covers when the reason is legitimate.

For everyone else it is a thirty-second check with an outsized downside if it is wrong. Somebody in your organisation, or an agency you parted with, may have flipped it during the rollout and told nobody.

Step 1: Make sure AI crawlers can access your content

Before any AI system can cite your website, its crawler needs to be able to read your pages. This is the most basic requirement and the one most often broken without anyone noticing.

Check your robots.txt. Open yourdomain.com/robots.txt and look for directives that mention AI user agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. If any are set to Disallow: /, those AI systems can't access your content at all. You can allow AI crawlers for search while still blocking them for training using the robots-ai.txt specification and ai.txt permission declarations.

Check your CDN and hosting settings. Cloudflare, Sucuri, and other CDN providers may block AI crawlers at the network level, before your robots.txt is even read. Review your firewall rules and bot management settings. Our technical visibility checklist walks through every common barrier and how to fix each one.

Check your server response. AI crawlers expect fast, clean HTML responses. Pages that rely heavily on client-side JavaScript rendering, return soft 404s, or require authentication will be skipped. Ensure your key pages return server-rendered HTML with proper status codes.

What being blocked looks like in practice

A specialist electronics manufacturer bought our Service Pack in July 2026, and their order stalled at the crawl stage. The reason was in their robots.txt: it allowed exactly three named crawlers (googlebot, bingbot and Slurp), then closed with User-agent: * and Disallow: /. A web application firewall in front of the site hung any request that didn't come from those three. Our crawler reached 1 page out of 15. They had 0 of 10 AI Discovery Files.

Their domains have been registered since the early 1990s and they rank perfectly well in classic search, which is exactly why nobody had questioned the setup. The owner confirmed by email that the blocking was deliberate: "For reasons I'm sure you would understand we disallow all agents and instead submit our sitemaps directly to google. This has worked out well as far as traditional search is concerned." In the same thread he described his goal for the year as "increasing the odds of being in the list of companies that are contained in a typical Google AI search," and said he was frustrated that AI results "identify companies that do not match the users interest."

Both things were true at once, and that is the point. Ranking on Google tells you nothing about whether AI systems can read you. We wrote that case up in full in blocking every crawler and expecting AI results. If any of it sounds familiar, check your own file before you read further.

The cost of blocking AI crawlers is measurable. Publishers who block them have seen a 23% decline in monthly visits overall, not just from AI platforms. As AI-powered search features become embedded in traditional search results, blocking AI crawlers can hurt your visibility across the board.

Which AI crawlers should you allow?

Not all AI crawlers do the same job, and blocking the wrong one carries very different costs. Some collect training data for future models. Others build search indexes or fetch your pages live while a user is mid-conversation. The table below lists every major AI crawler, who operates it, what it does, and what you lose by blocking it.

Two rows in that table catch people out. First, Google AI Overviews use Googlebot, not Google-Extended. Blocking Google-Extended keeps you out of Gemini's grounding but does nothing to remove you from AI Overviews; until this summer the only way to leave AI Overviews was to block Googlebot, which removed you from Google Search as well. That is no longer the trade: the Search generative AI control in Search Console now does it precisely, and Google says the setting is not used as a ranking signal in ordinary web search. Second, OpenAI splits search from training: OAI-SearchBot handles ChatGPT search while GPTBot collects training data. You can block GPTBot and still appear in ChatGPT search results, which is the right call for some publishers.

That distinction matters more than it looks, because most sites get it backwards. Across the same 1,744-site crawl, GPTBot was blocked on 9.9% of sites while ChatGPT-User, the crawler that fetches your page live for a user who is mid-conversation, was blocked on only 4.8%. ClaudeBot and Claude-User split the same way, 10% against 4.3%. Some of that is deliberate policy. Some of it is a copied blocklist that nobody has revisited. We pulled that data apart in the blocking guide.

If you want AI search visibility, this is the robots.txt starting point:

# robots.txt: allow AI search and assistant crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Around 21% of the top 1,000 websites actively block GPTBot through robots.txt, and Cloudflare's default now affects roughly 20% of the public web. Our own sample is broader than the top 1,000 and lands lower, at 9.9%, which is what you would expect: the largest publishers block hardest, and the long tail mostly hasn't decided anything.

How to get your content and code indexed by AI search engines

To get content or code indexed by AI search engines, it has to clear three bars in this order: the crawler must be allowed in, the words must exist in the raw HTML before any JavaScript runs, and for Google's AI surfaces the page must already be indexed in ordinary Google Search. Google states the third one flatly in its own guidance: To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements.

Read that sentence twice, because two obligations are hiding in it. Indexed, and eligible to be shown with a snippet. Most people notice the first and miss the second.

The snippet directive that quietly removes you from AI answers

When AI Overviews arrived, a lot of publishers reached for the only lever available and added a snippet restriction to keep their text out of the summary box. Those directives are still sitting in the markup of sites that have since changed their minds. nosnippet, max-snippet, data-nosnippet and noindex all limit what Google may show, and the same controls apply to AI Overviews and AI Mode. A page carrying nosnippet is not eligible to be a supporting link in either, however good the content is.

This is worth checking on any site that was actively managing its AI exposure in 2025, on any site that inherited a theme or an SEO plugin configuration from someone else, and on any site where a well-meaning developer copied a directive from a blog post. Grep your templates and your plugin settings for all four:

# Look for snippet restrictions in the served HTML, not just in your templates
curl -sL -A "Googlebot" https://yourdomain.com/ \
  | grep -oE 'name="robots"[^>]*|data-nosnippet'

# Check the HTTP header form too, which templates never show you
curl -sI -A "Googlebot" https://yourdomain.com/ | grep -i "x-robots-tag"

An empty result from both is what you want. If max-snippet comes back with a small number, you have capped how much of your page Google may quote, which caps how useful you can be to an answer that works by quoting. And note the second command: X-Robots-Tag is set at the server or CDN, so it will never appear in a template search. We have found it twice on client sites where nobody in the building knew it was there.

Since 31 August 2026 there is a cleaner way to make this decision. The Search generative AI control in Search Console covers AI features specifically, without touching your snippets in ordinary search results. If your goal was ever to stay out of AI answers while keeping normal search snippets, that switch is the right tool and the directives are the wrong one.

Serve the answer in the HTML, not after it

The second bar is the one that catches modern builds. Retrieval crawlers fetch, read and move on. Most do not execute JavaScript, and the ones that do give it no guarantees. If your headline, your prices, your service list or your location arrive after a client-side render, a crawler sees an empty shell and indexes nothing worth citing.

The test is one line and it is in the checking section below: fetch your own page with a crawler user agent and grep the raw response for a phrase you know is on it. If the phrase is missing, the content is not there to be indexed. Framework choice is not the problem; server-side rendering or static generation solves it in every mainstream framework.

Two smaller traps sit alongside it. Soft 404s, where a missing page returns 200 with an error message, teach a crawler that your error pages are content. And anything behind a login or an interstitial is simply not in the index, which is why gated whitepapers never get cited no matter how good they are. If you want a claim quoted, publish the claim.

Documentation and code have their own route now

If what you want indexed is developer documentation or code, the mechanics changed on 10 August 2026, when the llms.txt proposal moved to version 2. Version 2 dropped the context-expansion tooling that never got used and added something more practical: two link relations that let a page point a machine at a cleaner copy of itself.

<!-- In the <head> of an HTML page -->
<link rel="alternate" type="text/markdown" href="/docs/getting-started.md">
<link rel="describedby" href="/llms.txt">

rel="alternate" type="text/markdown" names a Markdown version of this page. rel="describedby" points at the llms.txt file that covers it. Both can be sent as HTTP Link headers instead, which is useful when you do not control the page template. Version 2 also settled a question the first version left open: a file covers the pages beneath its path, and the most specific file wins, so a team that only controls /docs/ can publish /docs/llms.txt and have it apply there.

The reason this matters for code specifically is that the heaviest consumers of llms.txt are not search engines. They are coding assistants. Cursor, Claude Code, Copilot and the MCP servers behind them fetch documentation directly while a developer is working, and a Markdown copy costs them a fraction of the tokens an HTML page does. If your product has an API, that is a real audience reading your docs today, and it is a different audience from the one arriving through AI Overviews. We come back to that distinction, and the evidence for it, in the section on whether anything reads these files.

Step 2: Publish AI Discovery Files so AI knows who you are

Access alone isn't enough. Once an AI crawler can read your pages, it needs to understand who you are, what you do, and how to represent you accurately. Without clear signals, you're leaving that interpretation to chance. This is the step that separates the 75.5% of sites sitting silent in the Passive tier from the 2.5% that are AI-Ready.

Sullivan's advice sounds simple, and that's what makes it easy to underestimate. "Write for humans" doesn't mean you can ignore machines. It means the best content for AI search is the same content that serves human readers well: clear, well-structured, and honest about who wrote it and why. What tripped me up the first time I heard this was the implication hiding underneath. If your content is good enough for humans but your site doesn't tell machines who you are, you're still invisible. The writing part is table stakes. The identity part is the gap most people miss.

Start with llms.txt

llms.txt is a plain-text file at your domain root that summarises your business for large language models. It is the single highest-value file to publish first, for two reasons. It is the one AI Discovery File with meaningful adoption already (7% of the sites in our crawl had one, against under 0.2% for every other file type), and it is the one Google checks.

That second point is recent and worth sitting with. In May 2026 Google added an Agentic Browsing category to Lighthouse, the engine behind PageSpeed Insights. One of its three default checks is the presence and validity of an llms.txt file at the domain root. Google is not endorsing the format as a ranking factor, and says so plainly: the category is still marked as under development. But it is now testing for the file in its own developer tooling, which is a long way from where the conversation was a year ago.

We ran this site through it on 17 June 2026 and scored Agentic Browsing 3/3, alongside Performance 98, Accessibility 100, Best Practices 100 and SEO 100 (mobile, emulated Moto G Power on Slow 4G, Lighthouse 13.4.0). That result is reproducible: put ai-visibility.org.uk into PageSpeed Insights yourself and check.

Here is a complete, working example for a small business, conforming to the llms.txt specification. Copy the shape, not the content:

# Meridian Plumbing

Lang: en-GB

> Gas boiler repair, replacement and servicing for domestic properties in
> Northamptonshire and south Leicestershire. Gas Safe registered, established
> 2009, based in Kettering, United Kingdom.

## About

Meridian Plumbing is a family-run heating firm founded in 2009 and based in
Kettering, Northamptonshire. We work exclusively on domestic gas boilers for
homeowners and landlords across Northamptonshire and south Leicestershire.
All engineers are Gas Safe registered (registration 123456).

## Services

- [Boiler repair](https://example.co.uk/boiler-repair/): Same-day diagnosis and repair for domestic gas boilers.
- [Boiler installation](https://example.co.uk/boiler-installation/): Replacements and new installations, all major brands.
- [Annual servicing](https://example.co.uk/servicing/): Yearly services and landlord gas safety certificates.

## Contact

- General enquiries: hello@example.co.uk
- Phone: +44 1536 555 0123
- Address: 14 Rockingham Road, Kettering, NN16 8JT, United Kingdom
- Service area: [Towns, counties and postcodes covered](https://example.co.uk/areas/)

## What We Do Not Do

Meridian Plumbing explicitly does not provide:
- Commercial catering equipment servicing
- Oil-fired or LPG boiler work
- Electrical work or general plumbing outside heating systems
- Any work outside the stated service area

## AI Discovery Files

- [AI Interaction Guidance](https://example.co.uk/ai.txt): Permissions and restrictions for AI systems
- [Brand Guidelines](https://example.co.uk/brand.txt): How to correctly reference our brand
- [Identity Data](https://example.co.uk/identity.json): Structured business identity information

# ---
# Specification: [llms.txt (ADF-001)](https://www.ai-visibility.org.uk/specifications/llms-txt/)

Four details in there are easy to skip and worth keeping. Lang: en-GB is the BCP 47 language declaration, which tells a machine this is British English before it starts guessing from spelling. About and Contact are required sections, not optional ones, and About should be a description rather than a list of links. The closing two lines are the attribution footer that names the specification the file follows; in our Q3 crawl, not one of the 164 sites publishing an AI Discovery File included any attribution, so a validator has no way of knowing which rules the file was written against.

The "What We Do Not Do" section does more work than the rest of the file put together. Stating what you don't do is the cheapest way to stop an AI system guessing, and guessing is where hallucinated business descriptions come from. If you only ever write one AI Discovery File, write this one, and include the exclusions. Our step-by-step llms.txt guide covers the format in full, and there is a complete worked example file set you can adapt.

Add identity.json and brand.txt

llms.txt is prose. Two other files carry the parts that need to be unambiguous. identity.json holds structured identity data: legal name, trading names, registration numbers, locations, and the services you provide, in a form that leaves no room for interpretation. brand.txt declares naming and terminology rules, which matters more than it sounds if your brand name is a common word, is easily confused with another company, or is routinely misspelled.

The Quick Start guide prioritises which files to create in which order, and the examples page has a complete set you can adapt. If you would rather not hand-write any of it, the free AI Discovery Files plugin for WordPress generates and serves all ten from your existing site content.

Then add Schema.org structured data

Schema.org and AI Discovery Files answer different questions, and you need both. Schema.org describes this page. AI Discovery Files describe this organisation. At minimum your site should carry Organization or LocalBusiness schema with your name, description, address, contact details, and sameAs links to your social profiles and directory listings. Research suggests sites with proper schema markup are 2-3 times more likely to appear in AI-generated answers. Entity disambiguation properties (sameAs, knowsAbout) are particularly valuable, because they let AI systems confirm your identity against sources you don't control.

Only 26.3% of the sites in our Q3 crawl had any Schema.org markup on the homepage, so this is still a differentiator rather than a baseline. We set out where the two standards overlap and where they don't in AI Discovery Files vs web standards.

Then check your existing content actually says what you do

AI systems that use RAG don't match keywords; they match meaning. A page about "affordable web design for small businesses" can be retrieved for a query like "who builds websites for startups on a budget?" even without those exact words. What matters is that your content clearly and completely describes what you offer, who you serve, and where you operate. If that information is vague, buried in marketing copy, or spread across dozens of pages without a clear summary, AI systems will struggle to extract it.

Does anything actually read these files?

It is the obvious objection, and it deserves a straight answer rather than a marketing one. We have measured it on our own property rather than guessing.

We installed our AI Discovery Files plugin on mcneece.com, published all ten files, and logged every request to them. Here is the dashboard for the 30 days to 25 July 2026, straight out of the plugin.

• 47File accesses
• 10Unique AI bots
• PerplexityBotMost active
• 0Blocked by robots.txt

47 file accesses from 10 distinct AI crawlers, and nothing blocked. The most active crawler over the window was PerplexityBot. Seven operators appeared: Perplexity, Meta, ByteDance, Anthropic, OpenAI, Microsoft and Apple. Here is every one of them, with the file each asked for most.

Now look at which crawlers did the reading, because this is the part that changes what you should do about it. GPTBot and ClaudeBot, the two training crawlers that get blocked most often, logged one access each. The work was done by the search and live-browsing crawlers: PerplexityBot on 11, Claude-User on 7, OAI-SearchBot on 6, bingbot on 4.

That is the argument for the crawler table above, in data. If you block GPTBot and ClaudeBot on principle, you have blocked the two crawlers that barely turn up. If you block OAI-SearchBot, Claude-User, ChatGPT-User or PerplexityBot, you have blocked the ones that came looking for who you are, repeatedly, while a user was mid-conversation.

The "top file" column holds a smaller surprise. OAI-SearchBot's most-requested file was not llms.txt. It was brand.txt, the naming and terminology file nobody writes about. PerplexityBot and Bytespider both went for developer-ai.txt. Meanwhile llm.txt, the deprecated singular variant, has still never been requested once. Publish the files crawlers actually ask for, and don't assume you know which those are.

Now the honest counterweight, because one site is one site and 47 reads in a month is not a flood. The evidence against this being an AI search tactic is substantial, and it has got stronger, not weaker, since we first wrote this section.

Ahrefs studied 137,000 domains and found that 97% of valid llms.txt files were never fetched at all. SE Ranking looked at roughly 300,000 domains and found no citation lift. The sharpest number comes from Limy, which monitored 515,382,577 AI bot events across a 90-day window and counted 408 requests to /llms.txt. That is not a small share. It is eight requests in every ten million events, from crawlers that were overwhelmingly fetching ordinary HTML instead.

And Google says so directly. Its guidance on optimising for generative AI features, updated on 10 July 2026, could not be plainer about its own systems.

The same document adds that structured data isn't required for generative AI search, and there's no special schema.org markup you need to add. If your only goal is Google's AI Overviews and AI Mode, take Google at its word and spend the afternoon on your content instead. We would rather say that plainly than sell you a file you do not need.

But read the next sentence in Google's own guidance, because it is the one nobody quotes: It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Google is not saying the files are worthless. It is saying Google Search does not read them, and pointing at the other systems that do.

That is the whole argument, and it is narrower than the way this gets sold. AI Discovery Files are not a Google ranking lever and never were. They are how you state your identity to every machine that is not Google: the assistant fetching your page mid-conversation, the coding agent reading your docs, the procurement bot checking whether you serve its postcode. Limy reached the same conclusion from the opposite direction, calling llms.txt a business-to-agent play rather than an SEO play, and recommending publishing it anyway.

So what do our 47 reads mean? Less than a 137,000-site aggregate, and more than nothing. Most llms.txt files in the wild sit on abandoned or malformed pages that nothing links to, so of course they go unread. A file that is valid, complete, linked and kept current gets crawled, and ours was crawled by ten distinct bots in a month. That is depth on one site against breadth across the web, and we are not going to pretend it overturns the aggregate. The full log data is published here, including the earlier window when the big LLM crawlers had not shown up at all. Judge the claim on the logs, not on our enthusiasm.

Step 3: Build credibility AI systems can verify

AI systems don't just find information. They evaluate whether to trust it. A page might answer the question perfectly, but if the AI can't verify the source, it may choose a less complete answer from a more credible site instead.

How to gain credibility on search engines for human and AI searches

You gain credibility for human and AI searches with the same work, because both audiences check the same things. Name the author. Publish a real address and a real phone number. Date the page and say when it was last reviewed. State your registration or accreditation numbers. Then make sure at least one source you do not control says the same thing. That list is not two strategies. It is one strategy read by two kinds of reader.

People ask us about this as though it were two projects, and it is the single most common framing mistake we see. The signals Google's quality raters are trained to look for (a named author with relevant experience, a verifiable organisation behind the page, a stated review date, corroboration from outside your own domain) are the same signals a retrieval system weighs when deciding whether a claim can safely be repeated in an answer. An AI system quoting you is staking its own accuracy on you. It behaves like a cautious editor, because that is effectively the job.

Where the two audiences genuinely differ is tolerance for ambiguity. A human reader who finds your address in the footer and a different one on your contact page will shrug and pick one. A machine cannot shrug. It lowers confidence in the whole source and may reach for a competitor who is merely consistent. That asymmetry is why the rest of this section is about consistency and corroboration rather than about writing better.

Internal consistency

Your business name, services, address, and contact details should say the same thing everywhere on your site: in the header, the footer, the About page, the Schema.org markup, and your AI Discovery Files. Contradictions between these sources create ambiguity, and AI systems handle ambiguity by reducing confidence in the source. The interoperability specification explains how AI Discovery Files relate to each other and how to avoid conflicts.

External corroboration

AI systems cross-reference your claims against third-party sources. If your website says you're a web design agency in Kettering, but Google Business Profile says you're in London, and your LinkedIn says you do "digital marketing," the AI can't resolve the contradiction. Keep your identity consistent across every public profile you control, and give AI systems as many independent places to check as you reasonably can.

Link your identity, don't invent it

The strongest corroboration is a link to an entity someone else maintains. A worked example from July 2026: we were doing an E-E-A-T pass on the site of a one-person executive chauffeur firm in Lincolnshire, and the owner turned out to be a Dutch former Football League goalkeeper with an English Wikipedia article and a Wikidata entity (Q391467). His Person node on the About page had no external identity links at all.

Two verified sameAs URLs fixed that. The business now ties its owner to an independent, encyclopaedic entity that a competing local firm structurally cannot replicate, and it took a schema edit rather than a marketing campaign. The full build is documented by Press Forge.

What we deliberately did not do matters as much. An AI-generated biography of him existed online, and it would have been easy to lift the football-to-chauffeur story from it. We left that for the client to tell in his own words, because inventing a real person's history to feed a trust signal poisons the trust it was meant to earn. Link the identity. Don't write the biography.

You almost certainly do not have a Wikipedia article, and that is fine. The principle scales down: sameAs your Companies House record, your trade body's public member listing, your verified Google Business Profile, your LinkedIn page. Anything a machine can fetch and check against you. Verify each URL resolves before you publish it, because a broken or wrong sameAs is worse than none.

Corroboration on its own is not sufficient either, and it is worth saying so. We took on a Staffordshire removals firm in June 2026 with a genuinely outstanding external record: 74 Google reviews at a 5.0 average, Trustpilot 4.8, and 133 reviews on a trade review site. It had not ranked on page one for any meaningful head term in three years of trading, because the site itself was roughly 160 near-identical templated town pages with nothing specific on them. The reviews were real and the site had nothing to attach them to. Once it was rebuilt around ten hand-written area pages and a full set of discovery files, it reached Complete conformance in our own directory and ChatGPT independently rated it 92/100. We wrote that up as the removals case study.

Fishkin's research ran 2,961 prompts across ChatGPT, Claude, and Google AI, asking for brand recommendations across 12 categories. The lists changed almost every time. When I first saw these numbers, my gut reaction was discouragement: if AI recommendations are that volatile, what's the point of optimising? But the same research found that visibility percentage (how often a brand appears across repeated runs) is statistically meaningful. Some brands showed up almost every time. Others barely appeared at all. The randomness is in the ordering, not in who makes the list. Clear identity signals, external corroboration, and structured data are what separate the brands that reliably appear from those that don't.

How to get cited by AI search engines

Being crawled makes you eligible. Getting featured in AI search results is a separate question, and it comes down to whether a passage on your page answers the question on its own, without the surrounding page to prop it up.

Think about what actually happens during retrieval. The system pulls a handful of candidate passages from across the web, drops them into a context window, and generates an answer from what it has. Your paragraph arrives stripped of its heading hierarchy, its sidebar, your logo, and everything a human reader would have used to work out who you are. If the passage doesn't stand up alone, it doesn't get used.

Google's own guidance puts the content bar first, ahead of every technical consideration, and words it more bluntly than most agencies would dare: Don't just recycle what others on the internet have already said, or could easily be produced by a generative AI model. Sit with the second half of that sentence for a moment. The test is not whether your page is accurate or well written. It is whether a model could have generated it without you. A page that passes that test has something in it that only you could have published: your prices, your job records, your crawl data, your client's actual postcode. Everything below is technique for making that material easy to lift.

Put the answer first

The single highest-return edit is moving the answer to the top. Most business writing warms up first. Retrieval punishes that.

Here is the pattern we see constantly on service pages:

At Meridian Plumbing we pride ourselves on our commitment to
excellence. With decades of combined experience, our friendly team is
passionate about delivering outstanding service to customers
throughout the region.

Four lines, and a machine has learned nothing it could repeat. There is no location, no service, no scope, no verifiable claim. Now the same opening rewritten so it survives being lifted out:

Meridian Plumbing repairs, services and replaces domestic gas boilers
in Kettering and across Northamptonshire and south Leicestershire. We
are Gas Safe registered and have traded since 2009. Emergency callouts
run seven days a week; we do not work on commercial catering equipment
or oil-fired systems.

Same length. The second version can be quoted directly in an answer and still be accurate and useful. That is the whole test.

Write headings the way people ask

A heading that reads "Our Approach" tells a retrieval system nothing. A heading that reads "How much does a boiler replacement cost in Northamptonshire?" matches the shape of a real question, and the passage under it becomes an obvious candidate. Clever headings cost you retrievals. This is the same reason we restructured this guide's own headings.

Give figures a named source and a date

AI systems weigh verifiability. "Studies show most websites aren't ready for AI" is unusable. "Our July 2026 crawl of 1,744 websites found that 90.6% published no AI Discovery File" can be repeated with attribution, which is exactly what a citation is. Name the source, give the date, and link it.

Keep sections self-contained

Avoid opening a section with "As mentioned above" or "This is why it matters." Those phrases are invisible glue for a human reader and dead weight for a retrieved passage. Each section should make sense to someone who has read nothing else on the page. Tables and short lists survive retrieval particularly well, because their structure carries meaning even when the formatting is stripped.

How to get found in AI search when nobody has heard of you

The most common objection at this point is that all of it favours big brands, and for classic search that is largely true. Retrieval is kinder. A system assembling an answer needs a passage that fits the question, and a small specific site can carry a better-fitting passage than a large general one. That is why a business with no backlinks and no domain authority can show up in AI search results ahead of an established competitor, provided the passage is specific enough to be worth quoting.

What this looks like when it works

The clearest evidence we have comes from a locksmith site we launched in April 2026. Instead of the usual templated town pages, it shipped 18 area pages each carrying real jobs from the owner's work diary: the customer's first name, the street, what had failed, what was fitted, and the actual price charged. Roughly 40 to 50 named job entries across the site, with prices from £30 to £249 stated plainly.

Within 10 to 12 days it was the #1 answer on Google, ChatGPT and Gemini for its head term, with zero backlinks acquired in that window. The part that matters for this section is what the AI engines did with it: asked about locksmiths in those towns, ChatGPT and Gemini quoted the specific named jobs and the real prices verbatim. Not paraphrased, not summarised. Lifted.

That is the mechanism in plain sight. Specific, self-contained, verifiable passages get reused because they can be. Generic ones cannot be quoted without saying nothing. The full case study has the detail, and the honest caveat that the build, the schema and the discovery files all contributed alongside the writing. We can't isolate one variable from a single site.

None of this guarantees a citation. Fishkin's data is a fair reminder of that. What it does is stop you losing citations you had already earned by making the passage unusable.

What doesn't work: five AI search myths

As AI search has grown, so has the volume of advice about how to "optimise" for it. Much of that advice is recycled from traditional SEO tactics that don't map to how AI systems actually work.

"Optimise your content for AI prompts." Some guides suggest writing content specifically designed to match common AI prompts. This is the AI equivalent of keyword stuffing. AI systems using RAG don't match prompts to pages; they match meaning. Write clearly about what you do and cover it thoroughly. That's it.

"Ask ChatGPT about your brand to see if you're visible." Testing prompts gives you a snapshot, not a strategy. Fishkin's research showed that AI recommendations are wildly inconsistent between runs. A single prompt test tells you almost nothing about your actual visibility. The definition of AI Visibility Checking draws a clear line between checking (validating your infrastructure) and tracking (monitoring AI outputs). Infrastructure is what you can control.

"Pay a tracker to monitor your AI brand mentions." The same problem, with a monthly invoice attached. If fewer than 1 in 100 identical prompts return the same list of brands, a dashboard charting that list week to week is charting noise. We went through the maths on this in why AI visibility trackers are usually a waste of money.

"Create AI-specific content pages." Building separate pages or Markdown versions of your content specifically for AI crawlers is fragile and unsustainable. These pages tend to go stale quickly and diverge from your actual content. The better approach is to make your existing pages machine-readable through structured data and AI Discovery Files, which sit alongside your content rather than duplicating it.

"Just focus on getting more backlinks." Backlinks matter for traditional search rankings, but AI systems using RAG prioritise content relevance and source trustworthiness over link authority. The locksmith site above reached #1 on ChatGPT and Gemini in under two weeks having acquired no backlinks at all in that window. Meanwhile the removals firm with 74 five-star reviews and links from three review platforms had spent three years off page one. Links still help pages get discovered, and third-party mentions still corroborate your identity, but they are not the lever they are in classic SEO.

How to check if your website is set up for AI search

"How can I check if my website is properly set up for AI search?" is the question we are asked most, and "can AI find my website?" is the same question asked more plainly. Both have a better answer than most people expect. Every step in this guide is verifiable. You do not have to take anyone's word for it, and you do not need a paid tool. Four checks, in this order, will tell you where you stand.

1. Read the Generative AI performance report in Search Console. This is the only check that tells you whether AI systems are already using you, rather than whether they could. Open Search Console, switch the Performance report to Generative AI, and look at the impression count. Then, in Settings, confirm the Search generative AI control is set to include your site. Both went worldwide on 31 August 2026, and the section above explains how to read the result and what the missing click column means. It covers Google's surfaces only, which is exactly why the next three checks exist.

2. Run the AI Visibility Checker. The free AI Visibility Checker fetches all ten AI Discovery File paths, validates what it finds, reads your robots.txt for crawler access, and looks for contradictions between the files. It returns a deterministic score with specific fixes, in under a minute. Deterministic matters here: run it twice and you get the same answer, which is not true of asking a chatbot about your brand.

3. Run PageSpeed Insights and read the Agentic Browsing result. Put your URL through PageSpeed Insights and look past Performance to the Agentic Browsing category. It reports a ratio rather than a score out of 100, and one of the three checks is your llms.txt. It is Google's own tooling telling you whether an agent can work with your page. We walk through what each of the three checks measures and how to pass all three.

4. Fetch your own site the way a crawler does. The checks above look at what you publish. This one looks at what actually gets served, which is not always the same thing once a CDN or firewall is involved:

# What an AI crawler receives, rather than what your browser receives
curl -sIL -A "OAI-SearchBot" https://yourdomain.com/ | grep -i "^HTTP\|^cf-\|^server"

# Confirm your llms.txt is served, and served as plain text
curl -sI -A "OAI-SearchBot" https://yourdomain.com/llms.txt | head -5

# Confirm your headline copy exists in the raw HTML, before any JavaScript runs
curl -sL -A "OAI-SearchBot" https://yourdomain.com/ | grep -o "Your Headline Text"

A 403 or 503 on the first command means something in front of your server is rejecting AI crawlers regardless of what robots.txt says. An empty result from the third means your content is rendered client-side and a crawler that doesn't execute JavaScript sees an empty page. Both are covered in the blocking guide.

If curl isn't your thing, the free 365i AI Crawler Checker does the same job in a browser. It tests 14 AI crawlers two ways, parsing your robots.txt and then sending live requests under each bot's user agent, and it writes you a message you can forward to your hosting provider if it finds something blocked at the server.

Finally, if you want to know whether AI crawlers are already visiting, the answer is in your server logs rather than in any tool. Search them for the user agents in the crawler table above. We published our own logs in full so you have something to compare against.

Log counting has one blind spot, and it is growing. Agentic browsers drive a real browser session, so ChatGPT Atlas, Perplexity Comet, Claude for Chrome and Opera Neon arrive looking like Chrome. The AI is present and the HTTP layer says nothing about it. Whatever your logs show for AI user agents, treat it as a floor rather than a total, and expect the gap between the two to widen as more people browse through an assistant.

Who can help you show up in AI search: doing it yourself, or paying someone

Not everyone wants to write JSON by hand, and a fair number of the people who ask us about this are looking for someone to keep their site and their listings consistent rather than a tutorial. If you are searching for who can help you appear in AI search results, or wondering whether an SEO agency can do this, here is the honest shape of the market: there are three routes, only one of them costs money, and the technical work is identical whichever you pick.

Be careful how you buy it. Plenty of agencies now sell an AI visibility retainer that is a monthly report of which prompts mentioned your brand. That is tracking, not fixing, and the prompts do not repeat reliably enough for the chart to mean much. We set out the arithmetic in why AI visibility trackers are usually a waste of money. Ask any prospective supplier a simpler question: what will be different on my website when you are finished, and how would I verify it myself?

Do it yourself. Every specification on this site is published in full, free, under CC BY 4.0, with worked examples for all ten files. The Quick Start guide is the shortest path: it tells you which two files to write first and what to put in them. Budget an afternoon.

Use the WordPress plugin. If your site runs WordPress, the free AI Discovery Files plugin generates and serves all ten files from your existing content, keeps them updated as your site changes, and logs which AI crawlers read them. That last part is the bit no manual approach gives you: you can see the bots arriving.

Have us write them. The AI Discovery Files Service Pack is our paid option. You complete an intake questionnaire about your business, and we write and validate the full set of ten files for your site. To be clear about what you get: we deliver the files ready to publish, and you or your developer upload them to your web root. We don't have access to your hosting, and we don't pretend to. If your site is on WordPress, the plugin does the publishing for you.

Whichever route you take, the work is the same work. The files either exist and agree with each other or they don't, and that is checkable by anyone, including you. Consistency across your listings and profiles is maintenance rather than a one-off project, so build a habit of updating your discovery files whenever your services, locations or contact details change.

AI search isn't replacing traditional search overnight. But it is absorbing a growing share of how people find, evaluate, and choose businesses. The websites that prepare for both channels now will be the ones that appear in both sets of results later. AI Visibility and SEO aren't competing priorities. They're parallel investments in being found accurately, everywhere that matters.

Frequently asked questions