One in Ten Top Websites Now Has
an AI Discovery File

AI Discovery File adoption passes 10%

For the first time since we started measuring, more than one in ten of the world's top websites publishes an AI Discovery File. Our crawl on 1 October 2026 found at least one of the ten files on 194 of 1,896 sites, or 10.2%, up from 9.4% in July and 6.5% in February.

Four things changed this quarter, and one surprised us:

• llms.txt got better as well as more common. Found on 150 sites (up from 122), valid and complete on 110 (up from 82), while invalid files stayed flat at 40.
• AI-Ready sites rose by more than a third, from 44 to 61, with Calendly, Check Point and the University of Exeter reaching it for the first time.
• Blanket AI blocking fell for the first time, from 23 sites to 17, while explicit permission rose from 31 to 38.
• Bluehost serves its llms.txt to GPTBot and ClaudeBot and refuses everyone else, our crawler included. That one result changes how every adoption figure, ours too, should be read.

The full Q4 2026 report has every number and the downloadable CSVs. This article is what we make of them.

What we measured in the Q4 2026 crawl

The crawler checked 1,995 domains: the top 1,000 globally from the Tranco research list and the top 1,000 UK domains, with the overlap removed. For each one it requested all ten AI Discovery Files at their canonical root URLs, validated what came back against the published specifications, and read robots.txt for 15 named AI user agents. The rules are on the methodology page, and nothing in the scoring is weighted by hand.

This was the cleanest crawl so far. Only 99 domains errored, against 251 in July and 535 in February, so the sample is 152 sites larger than last quarter and close to the full list. That makes the percentages more trustworthy. It also means some of this quarter's "new" adopters may simply be sites we couldn't reach in July, which is why the share matters more than the count here: 10.2% of a bigger, more complete sample is the stronger result.

Four quarters of AI Discovery File adoption

Over eight months the number of top sites publishing a file has doubled, and the number with a valid, complete llms.txt has trebled. The average readiness score hasn't moved off 2.2 in any quarter, though, because the three-quarters of sites in the Passive tier (reachable, publishing nothing) haven't moved either. Growth is happening at the top of the distribution, not across it.

The person who proposed llms.txt sees the same thing from the other end. When he revised the proposal in August, Jeremy Howard opened by describing what two years had produced:

"This is v2 of the proposal, updated based on what I learned from two years of adoption: thousands of sites publish an llms.txt file, documentation platforms generate one automatically, and Chrome's Lighthouse audits sites for one as part of its agentic browsing checks."

Jeremy Howard, author of the llms.txt proposal, in The /llms.txt file, v2, llmstxt.org, revised 10 August 2026 (verify quote at source)

What struck me reading that was the middle clause: documentation platforms generate one automatically. Our AI-Ready list leans heavily towards software companies (Datadog, Dynatrace, Cloudinary, ClassLink, Calendly), which is exactly the group most likely to have a docs platform writing the file for them. That's good for the adoption figures and a little sobering, because some of these files may reflect a vendor's default rather than a decision anyone at the business made. The Lighthouse point we've written about before, and it's the clearest sign yet that the file has moved from a proposal to something tools check for.

llms.txt adoption: more files, and better ones

The most encouraging number this quarter isn't the headline. llms.txt was found on 28 more sites than in July, and valid, complete files rose by exactly 28. Invalid files stayed at 40. In other words, every additional llms.txt we found this quarter was a good one, and the share that passes validation went from 67% to 73%.

Quality matters more than the count, because an invalid llms.txt can be worse than none: a model that reads a broken summary may repeat it with confidence. If you publish one, it's worth checking it against the llms.txt specification rather than assuming a generator got it right, and our step-by-step guide covers the common faults.

AI-Ready sites: who arrived this quarter, and who slipped

Tier 4 (AI-Ready) needs a valid AI Discovery File, a robots.txt that doesn't block AI crawlers, and Schema.org markup on the homepage. 61 sites made it this quarter, up from 44. Calendly, Check Point, Branch, hCaptcha and the University of Exeter reached it for the first time in our data. Dell and Greenpeace UK, both AI-Ready in April, are back after missing the July list.

Movement runs both ways, and we checked each departure individually rather than trusting the summary. Klaviyo, Life360 and the King's Fund are still AI-Ready; with 61 sites tied at the top, the published top 20 simply cuts off alphabetically before them. Bunkbedsstore and Groupon UK still publish a valid llms.txt but dropped to Tier 3. And Bluehost fell all the way to Passive, which turned out to be the most interesting result in the crawl.

The llms.txt only named AI crawlers can read

Bluehost was AI-Ready in July. In October our crawler got a 403 Forbidden for its llms.txt and recorded the file as missing. Before writing that up as "Bluehost removed its llms.txt", we asked for the same URL as five different visitors.

The file is there. Bluehost serves all 7,467 bytes of it to GPTBot and ClaudeBot and refuses a browser, a command-line client and our honestly labelled research crawler. That's an allowlist, almost certainly in bot protection in front of the site, and from the point of view of the AI crawlers it's working exactly as intended. From the point of view of every checker, auditor or researcher that isn't on the list, the file doesn't exist.

This is the problem a named-crawler allowlist is trying to solve, and the bot-protection vendors are explicit about what earns a crawler a place on one. Cloudflare's definition of a verified bot starts with honest self-identification:

"Honest self-identification: it declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS."

Cloudflare, Verified bots, Cloudflare Bot Solutions documentation, updated 1 July 2026 (verify quote at source)

I'll admit this one stung a little. Our crawler does exactly what that sentence asks: it says who it is, in its user agent, and links to the specification it checks against. It still gets turned away, because being honest about who you are is the first test and being on the list is the second, and a research crawler checking 2,000 sites four times a year will never be on anybody's list. I don't think Bluehost is wrong to do it. But it means our adoption figures are a floor, not a measurement of what the AI crawlers see, and we've added that to the methodology's known limitations today.

It also gives you a five-second test for your own site. If the two commands below return different status codes, your llms.txt is visible to some machines and invisible to others, and every tool that isn't on your list, including the AI Visibility Checker, will report it as missing:

# What an ordinary client gets
curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/llms.txt

# What OpenAI's crawler gets
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)" \
  https://www.example.com/llms.txt

Bluehost's sister brand HostGator Brasil fell from AI-Ready to Tier 3 for a related reason, except that its llms.txt returned 403 from our test machine whichever user agent we sent, GPTBot included. Whether that's a rule keyed to region or to network we can't tell from one location. Either way, a file that bot protection blocks for the crawlers it was written for isn't doing its job, and checking whether your site blocks AI crawlers should include checking the files, not only robots.txt.

AI crawler blocking falls, explicit permission rises

Sites that block every AI crawler in robots.txt fell from 23 to 17, the first drop in four quarters. Sites that explicitly allow them rose from 31 to 38. The two lines crossed back in April, and the gap has widened every quarter since.

Permission rose almost evenly, by nine or ten sites, for OpenAI's three agents, PerplexityBot and Google-Extended. The difference is on the blocking side, where the live-retrieval and Google agents lost blocks and the bulk crawlers didn't:

The pattern is a web making crawler-by-crawler choices rather than all-or-nothing ones, which is what the tooling now encourages. Google-Extended lost the most blocks (125 to 113) and ChatGPT-User lost four, while GPTBot and ClaudeBot were blocked as often as before and Diffbot gained 15 blocks. The large majority hasn't decided anything: 84.2% of sites have no AI-specific rule at all, exactly as in July.

Beyond llms.txt: first sightings of the other AI Discovery Files

For three quarters, nine of the ten files were effectively absent. This quarter identity.json, brand.txt and developer-ai.txt each turned up on a top site for the first time, and ai.txt and robots-ai.txt on two sites each.

None of the new ones passed validation. They're attempts rather than adoption, and llms.html went the wrong way: found on 49 sites, but valid files fell from 9 to 7 and complete ones from 3 to 0. Publishers are starting to look past llms.txt, and they're doing it without a template to work from. The quick-start guide and the example files exist for exactly that moment.

One figure hasn't moved at all. For the fourth quarter running, not one of the 194 sites publishing an AI Discovery File references a formal specification. Every file is still written from a blog post, a generator or memory.

AI Discovery Files compared with security.txt and ads.txt

Ten percent is easier to judge next to other root-level files that sites publish for machines rather than people:

security.txt has an IETF standard behind it (RFC 9116) and grew by 1.8 points this year. AI Discovery Files have no IETF standard and grew by 3.7. The comparison with established web standards is no longer a stretch, and neither is the case for treating these files as infrastructure rather than an SEO experiment.

What the Q4 2026 data cannot tell you

What the Q4 2026 data means for your website

If you run a business website, the useful conclusions are practical ones:

1. Publish an llms.txt, and validate it. One in ten top sites has an AI Discovery File; three in ten of the llms.txt files we find don't pass validation. Start with the specification, not a generator's defaults.
2. Test it as an AI crawler and as a stranger. Run the two curl commands above. If the answers differ, decide whether that's what you meant.
3. Make a deliberate robots.txt choice. 84% of top sites still haven't. Deciding crawler by crawler, as the sites in this report increasingly do, is better than inheriting a default.
4. Add the identity files next. identity.json and brand.txt are where the first tentative adopters are heading, and getting them valid now puts you ahead of almost every top site.

The AI Visibility Checker runs the same checks as this research against your own domain, free. If your site already publishes its files, you can submit it to the directory for verification, and the AI Visibility Checking definition explains why we measure inputs like these rather than chatbot mentions. For the hosting side, 365i can check whether bot protection on your site is hiding files you meant to publish.

Frequently asked questions