AI Discovery File Adoption Research

Key Findings

The first quarterly crawl establishes a baseline for AI Discovery File adoption across the web's most prominent domains. At 6.5%, adoption is nascent but already outpacing several established web standards at equivalent points in their lifecycle. The more pressing finding is not how few files exist, but how many of those that do exist are broken: over half fail validation. The web is not ignoring AI — it simply has no infrastructure for it yet.

1. Quality is the real gap, not awareness Of the 100 AI Discovery Files found across all domains, 56% are invalid — containing URL dumps, placeholder content, or malformed syntax. Only 36 files are complete and specification-compliant. Previous industry studies that concluded llms.txt had "zero measurable impact" never examined whether the files they found actually worked. Having a file and having it work are two different things.
2. llms.txt leads adoption but carries the highest error rate 55 domains serve an llms.txt file (3.8% of those crawled), making it the most adopted AI Discovery File by a wide margin. However, 20 of those files are invalid — a 36% failure rate. The llms.html companion file fares worse: 41 found, but only 7 valid and just 1 complete. Early adoption is concentrating around the files AI systems are most likely to request, but without validation, many of these files are actively misleading.
3. 87.5% of top websites have no AI crawler policy whatsoever The vast majority of websites have not yet addressed AI crawlers in their robots.txt. Of the 12.5% that have, most block selectively — targeting specific crawlers rather than setting a blanket policy. CCBot is the most frequently blocked agent (9.6%), followed by ClaudeBot (8.6%) and GPTBot (8.5%). Only 0.8% of sites explicitly allow AI access. The web has not decided how to handle AI crawlers; it has mostly not considered the question at all.
4. Nobody has reached Tier 5 — the ceiling is still open Zero domains in the sample achieved "AI-Optimised" status. The highest tier reached is Tier 4 (AI-Ready), occupied by just 22 domains — 1.5% of the sample. Meanwhile, 76.5% sit at Tier 2 (Passive), meaning they have a robots.txt but no AI-specific signals. Reaching Tier 4 today places a business ahead of 98.5% of the web's top domains. The competitive advantage for early movers is extraordinary, precisely because the bar is still so low.
5. Adoption is already outpacing comparable web standards At 6.5%, AI Discovery File adoption has already surpassed humans.txt (2.5%), a standard proposed in 2011. For context, robots.txt took roughly 15 years to reach 25% adoption; Schema.org took 8 years. ADF adoption appears to be on a significantly compressed timeline, likely driven by the commercial urgency of AI integration. If security.txt (12.7%, RFC published 2022) is any guide, ADF adoption could reach double digits within the next two to three quarters.
6. Early adopters span industries and geographies The 22 Tier 4 domains are not clustered in a single sector. They include global technology platforms (Shopify, Stripe, Opera), developer infrastructure (SourceForge, Dynatrace, Optimizely), UK public services (ScotRail, English Heritage, Energy Saving Trust), recruitment (Reed), and SaaS platforms (Mailchimp, Qualtrics, OneTrust). AI readiness is not a tech-sector concern — it is a cross-industry infrastructure decision.

— AI Visibility Research, February 2026