How to Appear in AI Search Results:
3 Steps That Actually Work

How to appear in AI search results: the quick answer

To appear in AI search results, your website needs three things. AI crawlers must be able to access your pages (check robots.txt, your CDN, and your firewall). Your identity must be machine-readable, which means publishing AI Discovery Files such as llms.txt and identity.json alongside Schema.org markup. And your business details must be consistent everywhere AI systems verify them, from your own site to Google Business Profile and LinkedIn.

We've watched these steps work from a standing start. A hand-built locksmith site launched in April 2026 with zero backlinks was the #1 answer on ChatGPT and Gemini for its head term within twelve days. The full case study documents exactly what was deployed. The rest of this guide explains why each step matters and how to carry it out on your own site.

Why AI search visibility matters now

ChatGPT now has over 900 million weekly active users. Google AI Overviews appear in roughly half of all search queries. Perplexity processes hundreds of millions of searches per month. AI isn't coming for traditional search. It's already absorbing it. For a breakdown of how these tools compare as consumer subscriptions, see which AI subscription is best for the average user.

For website owners, this creates a new problem. Your site might rank well on Google and still be completely invisible to the AI systems that are answering an increasing share of user questions. When someone asks ChatGPT "Who provides web design in Kettering?" or asks Gemini "What's the best CRM for small businesses?", the answer isn't a list of links. It's a direct, synthesised response that either includes your business or doesn't. We covered the small-business angle in detail in AI Visibility for Small Businesses: What It Actually Delivers, which uses the Lockerfella case as the worked example.

The numbers back this up. AI referral traffic grew 357% year-over-year in 2025. That traffic converts at five times the rate of traditional organic search. Visitors arriving from AI platforms spend 68% more time on site. The traffic is smaller than Google's, but it's higher quality and growing fast.

This guide covers what AI search engines actually need from your website, why most sites fail to provide it, and what you can do about it today.

How AI search engines find your website

AI search engines don't work like Google. Understanding the difference is the first step to appearing in their results.

Traditional search engines crawl your site, index its pages, and rank them against competing pages for specific queries. The output is a list of links. You compete for position.

AI search engines use a process called retrieval-augmented generation (RAG). When a user asks a question, the AI system searches for relevant web pages in real time, retrieves and reads them, then generates a synthesised answer. The output isn't a list. It's a paragraph (or several) with citations linking back to the sources used. You don't compete for position. You compete for inclusion. We break this down stage-by-stage in the full retrieval pipeline explainer.

ChatGPT triggers a web search on an estimated 20-35% of its prompts, which works out to 500-875 million web-retrieval queries per day. That's about one in eleven of Google's daily search volume, and it's growing.

What matters in this model isn't keyword density or backlink profiles. What matters is whether your content is accessible, clearly structured, and machine-readable. The AI needs to be able to crawl your page, understand what your business does, and trust the information enough to cite it.

Three things determine whether your website makes the cut: access, clarity, and trust. The three steps in this guide map directly onto them.

Why your website isn't showing up in AI search results

Here is the finding that surprises most people who ask us why they can't get found in AI search: you are probably not blocked. You are silent.

Every quarter our ADF adoption crawler fetches robots.txt and the ten AI Discovery File paths from roughly two thousand of the most visited websites in the world and the UK. The Q3 2026 crawl completed on 1 July 2026 and successfully read 1,744 sites. It scores each one into a readiness tier. The distribution is lopsided.

75.5% of sites sit in the Passive tier. AI crawlers can reach them perfectly well. There is simply nothing on the site that tells a machine who the business is, what it sells, or where it operates, beyond prose written for humans. Only 1.3% block every AI crawler outright. Deliberate blocking is real, and we cover it in the blocking checklist, but it is nowhere near the main reason businesses can't get found in AI search.

The supporting numbers point the same way. Across those 1,744 sites, 90.6% published no AI Discovery File of any kind, 73.7% had no Schema.org structured data on the homepage, and 84.2% had no AI crawler policy in robots.txt at all: no rule blocking them, no rule allowing them, nothing. Not one site in the sample reached the top AI-Optimised tier.

If your website isn't showing up in AI search results, work through these four causes in order. Each one has to be ruled out before the next one is worth investigating.

There is a fifth possibility worth naming honestly, because no amount of technical work removes it: the model may simply have picked someone else this time. Retrieval is not deterministic, and the same prompt run twice can return different sources. That is a genuine limit on what any of this can promise, and we come back to it in the myths section. What the four causes above give you is control over the part that is deterministic: whether you were eligible to be picked at all.

Step 1: Make sure AI crawlers can access your content

Before any AI system can cite your website, its crawler needs to be able to read your pages. This is the most basic requirement and the one most often broken without anyone noticing.

Check your robots.txt. Open yourdomain.com/robots.txt and look for directives that mention AI user agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. If any are set to Disallow: /, those AI systems can't access your content at all. You can allow AI crawlers for search while still blocking them for training using the robots-ai.txt specification and ai.txt permission declarations.

Check your CDN and hosting settings. Cloudflare, Sucuri, and other CDN providers may block AI crawlers at the network level, before your robots.txt is even read. Review your firewall rules and bot management settings. Our technical visibility checklist walks through every common barrier and how to fix each one.

Check your server response. AI crawlers expect fast, clean HTML responses. Pages that rely heavily on client-side JavaScript rendering, return soft 404s, or require authentication will be skipped. Ensure your key pages return server-rendered HTML with proper status codes.

What being blocked looks like in practice

A specialist electronics manufacturer bought our Service Pack in July 2026, and their order stalled at the crawl stage. The reason was in their robots.txt: it allowed exactly three named crawlers (googlebot, bingbot and Slurp), then closed with User-agent: * and Disallow: /. A web application firewall in front of the site hung any request that didn't come from those three. Our crawler reached 1 page out of 15. They had 0 of 10 AI Discovery Files.

Their domains have been registered since the early 1990s and they rank perfectly well in classic search, which is exactly why nobody had questioned the setup. The owner confirmed by email that the blocking was deliberate: "For reasons I'm sure you would understand we disallow all agents and instead submit our sitemaps directly to google. This has worked out well as far as traditional search is concerned." In the same thread he described his goal for the year as "increasing the odds of being in the list of companies that are contained in a typical Google AI search," and said he was frustrated that AI results "identify companies that do not match the users interest."

Both things were true at once, and that is the point. Ranking on Google tells you nothing about whether AI systems can read you. We wrote that case up in full in blocking every crawler and expecting AI results. If any of it sounds familiar, check your own file before you read further.

The cost of blocking AI crawlers is measurable. Publishers who block them have seen a 23% decline in monthly visits overall, not just from AI platforms. As AI-powered search features become embedded in traditional search results, blocking AI crawlers can hurt your visibility across the board.

Which AI crawlers should you allow?

Not all AI crawlers do the same job, and blocking the wrong one carries very different costs. Some collect training data for future models. Others build search indexes or fetch your pages live while a user is mid-conversation. The table below lists every major AI crawler, who operates it, what it does, and what you lose by blocking it.

Two rows in that table catch people out. First, Google AI Overviews use Googlebot, not Google-Extended. Blocking Google-Extended keeps you out of Gemini's grounding but does nothing to remove you from AI Overviews; only blocking Googlebot does that, and it removes you from Google Search entirely. Second, OpenAI splits search from training: OAI-SearchBot handles ChatGPT search while GPTBot collects training data. You can block GPTBot and still appear in ChatGPT search results, which is the right call for some publishers.

That distinction matters more than it looks, because most sites get it backwards. Across the same 1,744-site crawl, GPTBot was blocked on 9.9% of sites while ChatGPT-User, the crawler that fetches your page live for a user who is mid-conversation, was blocked on only 4.8%. ClaudeBot and Claude-User split the same way, 10% against 4.3%. Some of that is deliberate policy. Some of it is a copied blocklist that nobody has revisited. We pulled that data apart in the blocking guide.

If you want AI search visibility, this is the robots.txt starting point:

# robots.txt: allow AI search and assistant crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Around 21% of the top 1,000 websites actively block GPTBot through robots.txt, and Cloudflare's default now affects roughly 20% of the public web. Our own sample is broader than the top 1,000 and lands lower, at 9.9%, which is what you would expect: the largest publishers block hardest, and the long tail mostly hasn't decided anything.

Step 2: Publish AI Discovery Files so AI knows who you are

Access alone isn't enough. Once an AI crawler can read your pages, it needs to understand who you are, what you do, and how to represent you accurately. Without clear signals, you're leaving that interpretation to chance. This is the step that separates the 75.5% of sites sitting silent in the Passive tier from the 2.5% that are AI-Ready.

Sullivan's advice sounds simple, and that's what makes it easy to underestimate. "Write for humans" doesn't mean you can ignore machines. It means the best content for AI search is the same content that serves human readers well: clear, well-structured, and honest about who wrote it and why. What tripped me up the first time I heard this was the implication hiding underneath. If your content is good enough for humans but your site doesn't tell machines who you are, you're still invisible. The writing part is table stakes. The identity part is the gap most people miss.

Start with llms.txt

llms.txt is a plain-text file at your domain root that summarises your business for large language models. It is the single highest-value file to publish first, for two reasons. It is the one AI Discovery File with meaningful adoption already (7% of the sites in our crawl had one, against under 0.2% for every other file type), and it is the one Google checks.

That second point is recent and worth sitting with. In May 2026 Google added an Agentic Browsing category to Lighthouse, the engine behind PageSpeed Insights. One of its three default checks is the presence and validity of an llms.txt file at the domain root. Google is not endorsing the format as a ranking factor, and says so plainly: the category is still marked as under development. But it is now testing for the file in its own developer tooling, which is a long way from where the conversation was a year ago.

We ran this site through it on 17 June 2026 and scored Agentic Browsing 3/3, alongside Performance 98, Accessibility 100, Best Practices 100 and SEO 100 (mobile, emulated Moto G Power on Slow 4G, Lighthouse 13.4.0). That result is reproducible: put ai-visibility.org.uk into PageSpeed Insights yourself and check.

Here is a complete, working example for a small business, conforming to the llms.txt specification. Copy the shape, not the content:

# Meridian Plumbing

Lang: en-GB

> Gas boiler repair, replacement and servicing for domestic properties in
> Northamptonshire and south Leicestershire. Gas Safe registered, established
> 2009, based in Kettering, United Kingdom.

## About

Meridian Plumbing is a family-run heating firm founded in 2009 and based in
Kettering, Northamptonshire. We work exclusively on domestic gas boilers for
homeowners and landlords across Northamptonshire and south Leicestershire.
All engineers are Gas Safe registered (registration 123456).

## Services

- [Boiler repair](https://example.co.uk/boiler-repair/): Same-day diagnosis and repair for domestic gas boilers.
- [Boiler installation](https://example.co.uk/boiler-installation/): Replacements and new installations, all major brands.
- [Annual servicing](https://example.co.uk/servicing/): Yearly services and landlord gas safety certificates.

## Contact

- General enquiries: hello@example.co.uk
- Phone: +44 1536 555 0123
- Address: 14 Rockingham Road, Kettering, NN16 8JT, United Kingdom
- Service area: [Towns, counties and postcodes covered](https://example.co.uk/areas/)

## What We Do Not Do

Meridian Plumbing explicitly does not provide:
- Commercial catering equipment servicing
- Oil-fired or LPG boiler work
- Electrical work or general plumbing outside heating systems
- Any work outside the stated service area

## AI Discovery Files

- [AI Interaction Guidance](https://example.co.uk/ai.txt): Permissions and restrictions for AI systems
- [Brand Guidelines](https://example.co.uk/brand.txt): How to correctly reference our brand
- [Identity Data](https://example.co.uk/identity.json): Structured business identity information

# ---
# Specification: [llms.txt (ADF-001)](https://www.ai-visibility.org.uk/specifications/llms-txt/)

Four details in there are easy to skip and worth keeping. Lang: en-GB is the BCP 47 language declaration, which tells a machine this is British English before it starts guessing from spelling. About and Contact are required sections, not optional ones, and About should be a description rather than a list of links. The closing two lines are the attribution footer that names the specification the file follows; in our Q3 crawl, not one of the 164 sites publishing an AI Discovery File included any attribution, so a validator has no way of knowing which rules the file was written against.

The "What We Do Not Do" section does more work than the rest of the file put together. Stating what you don't do is the cheapest way to stop an AI system guessing, and guessing is where hallucinated business descriptions come from. If you only ever write one AI Discovery File, write this one, and include the exclusions. Our step-by-step llms.txt guide covers the format in full, and there is a complete worked example file set you can adapt.

Add identity.json and brand.txt

llms.txt is prose. Two other files carry the parts that need to be unambiguous. identity.json holds structured identity data: legal name, trading names, registration numbers, locations, and the services you provide, in a form that leaves no room for interpretation. brand.txt declares naming and terminology rules, which matters more than it sounds if your brand name is a common word, is easily confused with another company, or is routinely misspelled.

The Quick Start guide prioritises which files to create in which order, and the examples page has a complete set you can adapt. If you would rather not hand-write any of it, the free AI Discovery Files plugin for WordPress generates and serves all ten from your existing site content.

Then add Schema.org structured data

Schema.org and AI Discovery Files answer different questions, and you need both. Schema.org describes this page. AI Discovery Files describe this organisation. At minimum your site should carry Organization or LocalBusiness schema with your name, description, address, contact details, and sameAs links to your social profiles and directory listings. Research suggests sites with proper schema markup are 2-3 times more likely to appear in AI-generated answers. Entity disambiguation properties (sameAs, knowsAbout) are particularly valuable, because they let AI systems confirm your identity against sources you don't control.

Only 26.3% of the sites in our Q3 crawl had any Schema.org markup on the homepage, so this is still a differentiator rather than a baseline. We set out where the two standards overlap and where they don't in AI Discovery Files vs web standards.

Then check your existing content actually says what you do

AI systems that use RAG don't match keywords; they match meaning. A page about "affordable web design for small businesses" can be retrieved for a query like "who builds websites for startups on a budget?" even without those exact words. What matters is that your content clearly and completely describes what you offer, who you serve, and where you operate. If that information is vague, buried in marketing copy, or spread across dozens of pages without a clear summary, AI systems will struggle to extract it.

Does anything actually read these files?

It is the obvious objection, and it deserves a straight answer rather than a marketing one. We have measured it on our own property rather than guessing.

We installed our AI Discovery Files plugin on mcneece.com, published all ten files, and logged every request to them. Here is the dashboard for the 30 days to 25 July 2026, straight out of the plugin.

• 47File accesses
• 10Unique AI bots
• PerplexityBotMost active
• 0Blocked by robots.txt

47 file accesses from 10 distinct AI crawlers, and nothing blocked. The most active crawler over the window was PerplexityBot. Seven operators appeared: Perplexity, Meta, ByteDance, Anthropic, OpenAI, Microsoft and Apple. Here is every one of them, with the file each asked for most.

Now look at which crawlers did the reading, because this is the part that changes what you should do about it. GPTBot and ClaudeBot, the two training crawlers that get blocked most often, logged one access each. The work was done by the search and live-browsing crawlers: PerplexityBot on 11, Claude-User on 7, OAI-SearchBot on 6, bingbot on 4.

That is the argument for the crawler table above, in data. If you block GPTBot and ClaudeBot on principle, you have blocked the two crawlers that barely turn up. If you block OAI-SearchBot, Claude-User, ChatGPT-User or PerplexityBot, you have blocked the ones that came looking for who you are, repeatedly, while a user was mid-conversation.

The "top file" column holds a smaller surprise. OAI-SearchBot's most-requested file was not llms.txt. It was brand.txt, the naming and terminology file nobody writes about. PerplexityBot and Bytespider both went for developer-ai.txt. Meanwhile llm.txt, the deprecated singular variant, has still never been requested once. Publish the files crawlers actually ask for, and don't assume you know which those are.

Now the honest counterweight, because one site is one site and 47 reads in a month is not a flood. Ahrefs studied 137,000 domains and found that 97% of valid llms.txt files were never fetched at all. SE Ranking looked at roughly 300,000 domains and found no citation lift. Those studies measure breadth across the whole web; ours measures depth on one active, valid, complete set of files. Both can be true. Most llms.txt files in the wild sit on abandoned or malformed pages that nothing links to, so of course they go unread. A file that is valid, complete, linked and kept current gets crawled. Our data does not overturn a 137,000-site aggregate and we are not going to claim it does. The full log data is published here, including the earlier window when the big LLM crawlers had not shown up at all.

Step 3: Build credibility AI systems can verify

AI systems don't just find information. They evaluate whether to trust it. A page might answer the question perfectly, but if the AI can't verify the source, it may choose a less complete answer from a more credible site instead.

People often ask how to gain credibility for human and AI searches at the same time, as if they were two projects. They are one project with two audiences. The signals Google's quality raters look for (a named author, a real address, a stated review date, corroboration from outside your own domain) are the same signals an AI system uses to decide whether your claims can be repeated safely. Credibility, in both cases, comes from consistency and corroboration.

Internal consistency

Your business name, services, address, and contact details should say the same thing everywhere on your site: in the header, the footer, the About page, the Schema.org markup, and your AI Discovery Files. Contradictions between these sources create ambiguity, and AI systems handle ambiguity by reducing confidence in the source. The interoperability specification explains how AI Discovery Files relate to each other and how to avoid conflicts.

External corroboration

AI systems cross-reference your claims against third-party sources. If your website says you're a web design agency in Kettering, but Google Business Profile says you're in London, and your LinkedIn says you do "digital marketing," the AI can't resolve the contradiction. Keep your identity consistent across every public profile you control, and give AI systems as many independent places to check as you reasonably can.

Link your identity, don't invent it

The strongest corroboration is a link to an entity someone else maintains. A worked example from July 2026: we were doing an E-E-A-T pass on the site of a one-person executive chauffeur firm in Lincolnshire, and the owner turned out to be a Dutch former Football League goalkeeper with an English Wikipedia article and a Wikidata entity (Q391467). His Person node on the About page had no external identity links at all.

Two verified sameAs URLs fixed that. The business now ties its owner to an independent, encyclopaedic entity that a competing local firm structurally cannot replicate, and it took a schema edit rather than a marketing campaign. The full build is documented by Press Forge.

What we deliberately did not do matters as much. An AI-generated biography of him existed online, and it would have been easy to lift the football-to-chauffeur story from it. We left that for the client to tell in his own words, because inventing a real person's history to feed a trust signal poisons the trust it was meant to earn. Link the identity. Don't write the biography.

You almost certainly do not have a Wikipedia article, and that is fine. The principle scales down: sameAs your Companies House record, your trade body's public member listing, your verified Google Business Profile, your LinkedIn page. Anything a machine can fetch and check against you. Verify each URL resolves before you publish it, because a broken or wrong sameAs is worse than none.

Corroboration on its own is not sufficient either, and it is worth saying so. We took on a Staffordshire removals firm in June 2026 with a genuinely outstanding external record: 74 Google reviews at a 5.0 average, Trustpilot 4.8, and 133 reviews on a trade review site. It had not ranked on page one for any meaningful head term in three years of trading, because the site itself was roughly 160 near-identical templated town pages with nothing specific on them. The reviews were real and the site had nothing to attach them to. Once it was rebuilt around ten hand-written area pages and a full set of discovery files, it reached Complete conformance in our own directory and ChatGPT independently rated it 92/100. We wrote that up as the removals case study.

Fishkin's research ran 2,961 prompts across ChatGPT, Claude, and Google AI, asking for brand recommendations across 12 categories. The lists changed almost every time. When I first saw these numbers, my gut reaction was discouragement: if AI recommendations are that volatile, what's the point of optimising? But the same research found that visibility percentage (how often a brand appears across repeated runs) is statistically meaningful. Some brands showed up almost every time. Others barely appeared at all. The randomness is in the ordering, not in who makes the list. Clear identity signals, external corroboration, and structured data are what separate the brands that reliably appear from those that don't.

How to get cited by AI search engines

Being crawled makes you eligible. Getting featured in AI search results is a separate question, and it comes down to whether a passage on your page answers the question on its own, without the surrounding page to prop it up.

Think about what actually happens during retrieval. The system pulls a handful of candidate passages from across the web, drops them into a context window, and generates an answer from what it has. Your paragraph arrives stripped of its heading hierarchy, its sidebar, your logo, and everything a human reader would have used to work out who you are. If the passage doesn't stand up alone, it doesn't get used.

Put the answer first

The single highest-return edit is moving the answer to the top. Most business writing warms up first. Retrieval punishes that.

Here is the pattern we see constantly on service pages:

At Meridian Plumbing we pride ourselves on our commitment to
excellence. With decades of combined experience, our friendly team is
passionate about delivering outstanding service to customers
throughout the region.

Four lines, and a machine has learned nothing it could repeat. There is no location, no service, no scope, no verifiable claim. Now the same opening rewritten so it survives being lifted out:

Meridian Plumbing repairs, services and replaces domestic gas boilers
in Kettering and across Northamptonshire and south Leicestershire. We
are Gas Safe registered and have traded since 2009. Emergency callouts
run seven days a week; we do not work on commercial catering equipment
or oil-fired systems.

Same length. The second version can be quoted directly in an answer and still be accurate and useful. That is the whole test.

Write headings the way people ask

A heading that reads "Our Approach" tells a retrieval system nothing. A heading that reads "How much does a boiler replacement cost in Northamptonshire?" matches the shape of a real question, and the passage under it becomes an obvious candidate. Clever headings cost you retrievals. This is the same reason we restructured this guide's own headings.

Give figures a named source and a date

AI systems weigh verifiability. "Studies show most websites aren't ready for AI" is unusable. "Our July 2026 crawl of 1,744 websites found that 90.6% published no AI Discovery File" can be repeated with attribution, which is exactly what a citation is. Name the source, give the date, and link it.

Keep sections self-contained

Avoid opening a section with "As mentioned above" or "This is why it matters." Those phrases are invisible glue for a human reader and dead weight for a retrieved passage. Each section should make sense to someone who has read nothing else on the page. Tables and short lists survive retrieval particularly well, because their structure carries meaning even when the formatting is stripped.

What this looks like when it works

The clearest evidence we have comes from a locksmith site we launched in April 2026. Instead of the usual templated town pages, it shipped 18 area pages each carrying real jobs from the owner's work diary: the customer's first name, the street, what had failed, what was fitted, and the actual price charged. Roughly 40 to 50 named job entries across the site, with prices from £30 to £249 stated plainly.

Within 10 to 12 days it was the #1 answer on Google, ChatGPT and Gemini for its head term, with zero backlinks acquired in that window. The part that matters for this section is what the AI engines did with it: asked about locksmiths in those towns, ChatGPT and Gemini quoted the specific named jobs and the real prices verbatim. Not paraphrased, not summarised. Lifted.

That is the mechanism in plain sight. Specific, self-contained, verifiable passages get reused because they can be. Generic ones cannot be quoted without saying nothing. The full case study has the detail, and the honest caveat that the build, the schema and the discovery files all contributed alongside the writing. We can't isolate one variable from a single site.

None of this guarantees a citation. Fishkin's data is a fair reminder of that. What it does is stop you losing citations you had already earned by making the passage unusable.

What doesn't work: five AI search myths

As AI search has grown, so has the volume of advice about how to "optimise" for it. Much of that advice is recycled from traditional SEO tactics that don't map to how AI systems actually work.

"Optimise your content for AI prompts." Some guides suggest writing content specifically designed to match common AI prompts. This is the AI equivalent of keyword stuffing. AI systems using RAG don't match prompts to pages; they match meaning. Write clearly about what you do and cover it thoroughly. That's it.

"Ask ChatGPT about your brand to see if you're visible." Testing prompts gives you a snapshot, not a strategy. Fishkin's research showed that AI recommendations are wildly inconsistent between runs. A single prompt test tells you almost nothing about your actual visibility. The definition of AI Visibility Checking draws a clear line between checking (validating your infrastructure) and tracking (monitoring AI outputs). Infrastructure is what you can control.

"Pay a tracker to monitor your AI brand mentions." The same problem, with a monthly invoice attached. If fewer than 1 in 100 identical prompts return the same list of brands, a dashboard charting that list week to week is charting noise. We went through the maths on this in why AI visibility trackers are usually a waste of money.

"Create AI-specific content pages." Building separate pages or Markdown versions of your content specifically for AI crawlers is fragile and unsustainable. These pages tend to go stale quickly and diverge from your actual content. The better approach is to make your existing pages machine-readable through structured data and AI Discovery Files, which sit alongside your content rather than duplicating it.

"Just focus on getting more backlinks." Backlinks matter for traditional search rankings, but AI systems using RAG prioritise content relevance and source trustworthiness over link authority. The locksmith site above reached #1 on ChatGPT and Gemini in under two weeks having acquired no backlinks at all in that window. Meanwhile the removals firm with 74 five-star reviews and links from three review platforms had spent three years off page one. Links still help pages get discovered, and third-party mentions still corroborate your identity, but they are not the lever they are in classic SEO.

How to check if your website is set up for AI search

"How can I check if my website is properly set up for AI search?" is the question we are asked most, and it has a better answer than most people expect. Every step in this guide is verifiable. You do not have to take anyone's word for it, and you do not need a paid tool. Three checks, in this order, will tell you where you stand.

1. Run the AI Visibility Checker. The free AI Visibility Checker fetches all ten AI Discovery File paths, validates what it finds, reads your robots.txt for crawler access, and looks for contradictions between the files. It returns a deterministic score with specific fixes, in under a minute. Deterministic matters here: run it twice and you get the same answer, which is not true of asking a chatbot about your brand.

2. Run PageSpeed Insights and read the Agentic Browsing result. Put your URL through PageSpeed Insights and look past Performance to the Agentic Browsing category. It reports a ratio rather than a score out of 100, and one of the three checks is your llms.txt. It is Google's own tooling telling you whether an agent can work with your page. We walk through what each of the three checks measures and how to pass all three.

3. Fetch your own site the way a crawler does. The checks above look at what you publish. This one looks at what actually gets served, which is not always the same thing once a CDN or firewall is involved:

# What an AI crawler receives, rather than what your browser receives
curl -sIL -A "OAI-SearchBot" https://yourdomain.com/ | grep -i "^HTTP\|^cf-\|^server"

# Confirm your llms.txt is served, and served as plain text
curl -sI -A "OAI-SearchBot" https://yourdomain.com/llms.txt | head -5

# Confirm your headline copy exists in the raw HTML, before any JavaScript runs
curl -sL -A "OAI-SearchBot" https://yourdomain.com/ | grep -o "Your Headline Text"

A 403 or 503 on the first command means something in front of your server is rejecting AI crawlers regardless of what robots.txt says. An empty result from the third means your content is rendered client-side and a crawler that doesn't execute JavaScript sees an empty page. Both are covered in the blocking guide.

If curl isn't your thing, the free 365i AI Crawler Checker does the same job in a browser. It tests 14 AI crawlers two ways, parsing your robots.txt and then sending live requests under each bot's user agent, and it writes you a message you can forward to your hosting provider if it finds something blocked at the server.

Finally, if you want to know whether AI crawlers are already visiting, the answer is in your server logs rather than in any tool. Search them for the user agents in the crawler table above. We published our own logs in full so you have something to compare against.

Who can set your website up for AI search

Not everyone wants to write JSON by hand, and a fair number of the people who ask us about this are looking for someone to keep their site and their listings consistent rather than a tutorial. There are three honest routes, and only one of them costs money.

Do it yourself. Every specification on this site is published in full, free, under CC BY 4.0, with worked examples for all ten files. The Quick Start guide is the shortest path: it tells you which two files to write first and what to put in them. Budget an afternoon.

Use the WordPress plugin. If your site runs WordPress, the free AI Discovery Files plugin generates and serves all ten files from your existing content, keeps them updated as your site changes, and logs which AI crawlers read them. That last part is the bit no manual approach gives you: you can see the bots arriving.

Have us write them. The AI Discovery Files Service Pack is our paid option. You complete an intake questionnaire about your business, and we write and validate the full set of ten files for your site. To be clear about what you get: we deliver the files ready to publish, and you or your developer upload them to your web root. We don't have access to your hosting, and we don't pretend to. If your site is on WordPress, the plugin does the publishing for you.

Whichever route you take, the work is the same work. The files either exist and agree with each other or they don't, and that is checkable by anyone, including you. Consistency across your listings and profiles is maintenance rather than a one-off project, so build a habit of updating your discovery files whenever your services, locations or contact details change.

AI search isn't replacing traditional search overnight. But it is absorbing a growing share of how people find, evaluate, and choose businesses. The websites that prepare for both channels now will be the ones that appear in both sets of results later. AI Visibility and SEO aren't competing priorities. They're parallel investments in being found accurately, everywhere that matters.

Frequently asked questions