AI Discovery File Adoption Research

Key Findings

The Q2 2026 crawl shows meaningful growth across every dimension of AI readiness. ADF adoption rose from 6.5% to 7.2%, but the real story is in the infrastructure: 445 fewer domains errored this quarter, meaning our sample now covers 1,905 of 1,995 domains — a far more complete picture. Selective AI crawler blocking jumped 2 percentage points as more organisations make deliberate decisions about AI access, and the number of domains with any AI policy in robots.txt is growing faster than ADF file adoption itself.

1. llms.txt adoption accelerating — now the clear leader llms.txt grew from 55 to 93 domains (+69%), reaching 4.9% adoption. Valid files nearly doubled from 35 to 61, and complete implementations rose from 35 to 64. While the invalid count also grew (20 to 32), the ratio of valid-to-found improved from 64% to 66% — quality is keeping pace with volume for the first time.
2. Selective AI crawler blocking surges past 12% Domains blocking AI crawlers selectively jumped from 151 to 235 (+56%), now reaching 12.3% of crawled domains. Meanwhile, "no AI policy" dropped 2.5 percentage points to 85%. This suggests organisations are actively choosing which AI systems can access their content rather than ignoring the question. PerplexityBot saw the steepest blocking increase (+1.5pp), possibly reflecting growing awareness of AI search agents.
3. Crawl reliability transformed — error rate dropped from 27% to 5% Only 90 domains errored in Q2 versus 535 in Q1, bringing the success rate from 73% to 95%. This is an infrastructure improvement in the crawler itself, but it has a significant analytical impact: many previously unmeasured domains are now included, making the adoption and blocking figures more representative of actual web behaviour.
4. Tier 3 (Partially Ready) grows while Passive shrinks The share of Passive domains (Tier 2) dropped from 76.5% to 73.7%, with most movement going to Partially Ready (Tier 3, up from 19.5% to 21.7%). AI-Ready (Tier 4) edged up from 22 to 33 domains. Tier 5 remains at zero — no domain yet combines 3+ valid ADFs, explicit AI crawler permission, and Schema.org markup.
5. Major brands join the top adopters — NVIDIA, Dell, ASUS enter the list Nine new domains entered the top 20 adopters, including NVIDIA, Dell, ASUS, Datadog, and Cloudinary. Nine dropped out, including Stripe, Shopify, and SourceForge. The new entrants are predominantly enterprise technology companies, suggesting that AI readiness is becoming a priority in corporate web strategy rather than remaining a niche concern.
6. Schema.org adoption jumps to nearly 30% Schema.org presence on homepages rose from 25.6% to 29.7% (+4.1pp, 374 to 566 domains). This is the largest single-metric jump in the dataset and reflects the broader trend of websites investing in machine-readable signals. Since Schema.org is a prerequisite for the highest readiness tiers, this growth creates a foundation for future ADF tier upgrades.

— AI Visibility Research, April 2026