AI Discovery File Adoption Research

Key Findings

The Q4 2026 crawl puts AI Discovery File adoption above 10% for the first time: 194 of 1,896 successfully crawled domains (10.2%) now publish at least one file, up from 164 (9.4%) in Q3. llms.txt still carries almost all of it, but the files it carries are getting better, and the number of AI-Ready sites rose by more than a third. This quarter is also the cleanest comparison so far. Only 99 domains errored, against 251 in July, so the sample is 152 domains larger and much closer to the full list. That cuts both ways: the percentages are more trustworthy, and some of the new adopters may simply be sites we could not reach last time rather than sites that changed.

1. Adoption passes 10%, and the better crawl makes the rise easier to trust Domains with any AI Discovery File rose from 164 to 194 (+18%), and the share from 9.4% to 10.2%. The share matters more than the count here, because the crawl reached 1,896 domains this quarter against 1,744 in July (errors fell from 251 to 99). A rise in the percentage on a larger, more complete sample is the opposite of last quarter, when a shrinking sample flattered some numbers. We still cannot tell from aggregate data alone how many of the 30 new adopters were unreachable in July rather than newly published.
2. llms.txt grows, and its quality improves faster than its count llms.txt was found on 150 domains, up from 122 (+23%), while valid and complete files rose from 82 to 110 (+34%). Invalid llms.txt files stayed exactly flat at 40, so every additional file this quarter was a good one: the valid share rose from 67% to 73%. Across all file types the count of complete files went from 85 to 110. Publishers are not only adopting llms.txt, they are getting the format right more often.
3. AI-Ready sites up by more than a third, with Calendly, Check Point and Exeter reaching it for the first time Tier 4 (AI-Ready) domains rose from 44 to 61 (+39%), now 3.2% of the sample against 2.5% in Q3. Calendly, Check Point, Branch, hCaptcha and the University of Exeter reach AI-Ready for the first time in our data, and Dell and Greenpeace UK, both AI-Ready in Q2, are back after missing the July list. Movement runs both ways: Bunkbedsstore and Groupon UK still publish a valid llms.txt but fell to Tier 3, and Bluehost now returns 403 to our crawler while serving the same file to GPTBot and ClaudeBot (see the methodology note on user-agent gating). Tier 5 (AI-Optimised) is still empty after four quarters, and the average readiness score held at 2.2 out of 5.0.
4. Blanket AI blocking falls, explicit permission rises again Sites blocking every AI crawler fell from 23 to 17 (1.3% to 0.9%), the first real drop in four quarters, while sites explicitly allowing AI crawlers rose from 31 to 38 (2.0%). Selective blocking edged up from 217 to 239 domains (12.6%), so the trend is towards deliberate, per-crawler decisions rather than all-or-nothing. The large majority is unchanged: 84.2% of sites still have no AI policy at all, the same share as in July.
5. Other file types appear for the first time, but none of them is valid yet identity.json, brand.txt and developer-ai.txt were each found on a domain for the first time, and ai.txt and robots-ai.txt each on two. None of those new files passed validation, so they are attempts rather than adoption. llms.html moved the wrong way on quality: found on 49 domains (from 46), but valid files fell from 9 to 7 and complete files from 3 to 0. Breadth is starting, but only llms.txt is being implemented to a standard a machine can rely on.
6. Still zero of 194 adopters reference a formal specification For the fourth quarter running, not one domain publishing an AI Discovery File references a formal specification. For scale, AI Discovery Files are now on 10.2% of crawled sites against 14.5% for security.txt and 2.3% for humans.txt, so this is no longer a fringe practice, yet every one of these files is still written without a shared schema or version to validate against. Adoption keeps outrunning standardisation, and the gap widens as adoption grows.

AI Visibility Research, October 2026