AI Visibility Research
Quarterly AI Visibility Research tracking llms.txt, ai.txt, and AI Discovery File adoption across 1,905 top domains
How Many Websites Are AI-Ready?
Our AI Visibility Research programme tracks AI Discovery File adoption — including llms.txt, ai.txt, and identity.json — across the web's most prominent domains. Each quarter, we crawl the top 1,000 global and 1,000 UK domains to measure which sites have AI Discovery Files, whether those files pass structural validation, and how sites manage AI crawler access via robots.txt. All AI Visibility Research data is published under CC BY 4.0 with full methodology documentation.
AI Visibility Research: Adoption Over Time
How AI Discovery File adoption has changed across our quarterly crawls. This measures the percentage of crawled domains with at least one valid AI Discovery File.
View data table
| Quarter | Adoption | Avg Readiness | No AI Policy |
|---|---|---|---|
| Q1 2026 | 6.5% | 2.2 | 87.5% |
| Q2 2026 | 7.2% | 2.2 | 85.0% |
Adoption by File Type
Which AI Discovery Files are growing fastest? Each line tracks the adoption rate of a specific file type across all quarters.
View data table
| File | Q1 2026 | Q2 2026 |
|---|---|---|
| llms.txt | 3.8% | 4.9% |
| llms.html | 2.8% | 2.5% |
| ai.txt | 0.1% | 0.1% |
| ai.json | 0.1% | 0.1% |
| identity.json | 0.0% | 0.0% |
| brand.txt | 0.0% | 0.1% |
| faq-ai.txt | 0.0% | 0.1% |
| developer-ai.txt | 0.0% | 0.0% |
| robots-ai.txt | 0.1% | 0.1% |
Readiness Distribution Over Time
How the distribution of AI readiness tiers is shifting quarter over quarter. Readiness tiers range from 0 (Unaware) to 5 (AI-Optimised).
View data table
| Tier | Q1 2026 | Q2 2026 |
|---|---|---|
| AI-Optimised | 0.0% | 0.0% |
| AI-Ready | 1.5% | 1.7% |
| Partially Ready | 19.5% | 21.7% |
| Passive | 76.5% | 73.7% |
| Actively Blocking | 1.3% | 1.2% |
| Unaware | 1.2% | 1.7% |
AI Visibility Research Reports
Each quarter, our AI Visibility Research crawls the top global and UK domains to measure AI Discovery File adoption, quality, and crawler access policies.
Download AI Visibility Research Data
All AI Visibility Research data is available as downloadable CSV files under a CC BY 4.0 licence. Each quarterly report includes summary statistics, per-domain readiness scores, and AI crawler blocking data.
Frequently Asked Questions
Everything you need to know about our AI Visibility Research — methodology, data, and how to improve your score.
About the Research
What this study covers and how it works.
What are AI Discovery Files?
AI Discovery Files (ADFs) are a set of 10 standardised root-level files — including llms.txt, ai.txt, ai.json, identity.json, and brand.txt — that help AI systems discover, interpret, and correctly represent a website. They are the AI equivalent of robots.txt for search engines: machine-readable signals that define how a site should be understood by large language models like ChatGPT, Claude, and Gemini.
What does this research measure?
This research measures three things across the web's top domains: how many have AI Discovery Files and whether those files pass structural validation, how websites use robots.txt to manage AI crawler access, and each domain's overall AI readiness on a 0–5 tier scale. Data is collected by crawling approximately 2,000 of the web's most prominent domains quarterly.
How many websites are included in the crawl?
We crawl the Global Top 1,000 and UK Top 1,000 domains, sourced from the Tranco List — a research-grade domain ranking that aggregates multiple popularity lists to resist manipulation. After deduplication, the crawl covers approximately 1,995 unique domains each quarter.
How often is the crawl performed?
Crawls run quarterly on the 1st of January, April, July, and October. Results are typically published within one week of the crawl completing. Each quarter is archived as a standalone report with downloadable data.
Who runs this research?
This research is conducted by AI Visibility, the publisher of the AI Discovery Files specification. The study is independent, transparent, and published under CC BY 4.0 so anyone can verify or build on the findings.
Reading the Data
How to interpret adoption rates, quality scores, and readiness tiers.
How are AI Readiness Tiers calculated?
Each domain receives a tier from 0 to 5 based on three inputs: valid AI Discovery File count, AI crawler policy in robots.txt, and Schema.org presence on the homepage. Rules are evaluated in order — the first matching rule determines the tier. See the full scoring methodology for details.
What counts as a valid AI Discovery File?
A file is considered valid if it achieves a quality rating of "minimal" or "complete" after structural validation. "Minimal" means all required checks pass. "Complete" means all required and recommended fields are present. Files that fail required checks are rated "invalid".
How does AI Discovery File adoption compare to other web standards?
AI Discovery Files have already surpassed humans.txt (proposed 2011) and are approaching security.txt (RFC published 2022). For context, robots.txt took roughly 15 years to reach 25% adoption. AI Discovery File adoption appears to be on a compressed timeline, driven by the commercial urgency of AI integration.
What does the quality distribution show?
The quality distribution breaks down each AI Discovery File type by rating: "complete", "minimal", "invalid", and "not found". This shows not just whether sites have the files, but whether the files are well-formed and useful to AI systems.
AI Crawler Access
How websites manage AI crawler permissions via robots.txt.
Which AI crawlers are blocked most often?
CCBot (Common Crawl), ClaudeBot (Anthropic), and GPTBot (OpenAI) are consistently the most frequently blocked AI agents. Blocking patterns vary by industry — media sites tend to block more aggressively, while technology companies are more permissive.
Does blocking AI crawlers affect AI visibility?
Yes, directly. If your robots.txt blocks AI crawlers, those systems cannot access your content. However, blocking is a legitimate choice — particularly for paywalled or copyrighted content. Run a free check to see your current status.
What is the difference between blocking and having no policy?
Having no AI-specific policy means you haven't made a deliberate decision — AI crawlers access your site under general rules. Explicitly blocking means you've chosen to deny access. Both are valid positions, but no policy leaves your AI visibility to chance.
Using the Data
Downloads, licensing, and improving your own score.
Can I download the raw research data?
Yes. All crawl data is freely available as CSV downloads, licensed under CC BY 4.0. Three files are provided per quarter. Visit any quarterly report page to access its datasets.
How can my website improve its AI readiness?
Start by adding the core AI Discovery Files (llms.txt, ai.txt, identity.json) to your website root. Review your robots.txt to set an explicit AI crawler policy. Use the free AI Visibility Checker to assess your current state.
When is the next crawl scheduled?
Crawls run quarterly: 1st January, 1st April, 1st July, and 1st October. Results are typically published within one week.
AI Visibility Research Methodology
How We Collect This Data
Our crawler checks the top 1,000 global and top 1,000 UK domains (deduplicated to ~1,995) for all 10 AI Discovery Files, validates each against the specification, analyses robots.txt AI crawler policies across 15 known agents, and scores each domain's overall AI readiness using a deterministic tier model. The full methodology — including validation rules, soft 404 detection, redirect classification, and scoring logic — is published for transparency.