Research Methodology

This document describes how we collect, validate, and score AI Discovery File adoption data across the web's most prominent domains. We publish our methodology in full to enable scrutiny and reproducibility.

Domain Selection

We crawl two overlapping lists of prominent websites:

• Global Top 1,000 — The top 1,000 domains worldwide by traffic rank (sourced from the Tranco List, a research-grade domain ranking that aggregates multiple lists to resist manipulation).
• UK Top 1,000 — The top 1,000 domains popular in the United Kingdom.

The lists are combined and deduplicated. When a domain appears in both lists, the global entry takes precedence and the UK duplicate is discarded. After deduplication, the actual crawl covers approximately 1,995 unique domains. This selection captures the websites most likely to be referenced by AI systems when answering user questions.

Crawl Process

The crawler is a Cloudflare Worker that runs quarterly (January, April, July, October). Each crawl:

1. Fetches domain lists from KV storage, deduplicates (global list takes precedence), and splits domains into processing batches of 25.
2. For each domain, fetches robots.txt first, then all 10 AI Discovery Files sequentially at their canonical URLs (e.g., https://example.com/llms.txt, https://example.com/ai.json), with a 500ms delay between each request.
3. Validates each file against its specification — checking required fields, format compliance, and content quality (see File Validation).
4. Checks robots.txt for AI-specific crawler directives, testing 15 known AI user agents (see AI Crawler Analysis).
5. Fetches comparison files — security.txt, humans.txt, ads.txt — for adoption context.
6. Fetches the homepage and checks for Schema.org structured data (see Schema.org Detection).
7. Computes a readiness tier (0–5) for each domain using the combinatorial scoring model (see Readiness Scoring).
8. Aggregates results into a summary report once all batches complete.

Request Behaviour

The crawler identifies itself as ADF-Adoption-Crawler/1.0. All requests:

• Use a 10-second timeout per file fetch
• Follow up to 3 redirects (manually tracked — see Redirect Classification)
• Include a 500ms delay between consecutive requests to the same domain (10 files + 4 comparison files + homepage = minimum 7.5 seconds per domain)
• Process batches with 5 concurrent domains, further limiting aggregate request rate
• Truncate response bodies at 50KB per file and 100KB for homepage fetches to prevent memory exhaustion
• Respect robots.txt (the crawler checks for self-blocking and records it)

Files Checked

We check for all 10 AI Discovery Files defined in the ADF specification:

File Validation

Each file goes through a multi-stage validation pipeline. The crawler first determines whether the file genuinely exists (handling redirects and soft 404s), then validates its structure against the ADF specification.

Quality Tiers

Every ADF file is assigned one of four quality levels based on structural validation:

For readiness scoring purposes, valid means a quality of "minimal" or "complete" — files rated "invalid" do not count toward a domain's valid ADF total.

Structural Checks per File Type

Each ADF file type has its own set of required and recommended checks. For example:

• llms.txt (ADF-001) — Required: contains a # Heading, a > blockquote, ## subsections, and 50+ words. Recommended: 3+ subsections, contact info.
• ai.txt (ADF-004) — Required: [identity], [permissions], [restrictions] sections. Recommended: [attribution], [contact], [content-types].
• ai.json (ADF-005) — Required: valid JSON, name, url, permissions array, restrictions array. Recommended: $schema reference, attribution object.
• identity.json (ADF-006) — Required: valid JSON, name, type, url, description (20+ characters). Recommended: alternateNames, contactPoints.
• brand.txt (ADF-007) — Required: [official-names], [incorrect-names], [naming-rules]. Recommended: [brand-voice], [key-people], [quotation-policy].
• faq-ai.txt (ADF-008) — Required: Q: and A: pairs, at least 3 Q&A pairs. Recommended: category headings, answers averaging 50+ characters.

Full validation rules for all 10 file types are defined in the ADF specification.

Soft 404 Detection

Many web servers return an HTTP 200 status code for missing pages, serving a custom error page instead of a proper 404. The crawler applies six detection heuristics, checked in order:

1. Body too small — Response body under 50 bytes is treated as empty/placeholder.
2. Content-Type mismatch — A .txt or .json file returning text/html indicates a custom error page.
3. HTML body in non-HTML file — Body starts with <!doctype, <html, or <head when the file should be plain text or JSON.
4. Title tag contains 404 — The <title> element contains "404" or "not found".
5. Common error strings — The first 2,000 characters of the body are checked for 11 known error phrases: "page not found", "page could not be found", "nothing was found", "404 error", "error 404", "does not exist", "no longer available", and others.
6. WordPress catch-all — Non-HTML file under 1,000 bytes containing wp-content, indicating a WordPress theme's custom 404 response.

Files matching any heuristic are classified as "not found" rather than "invalid". This distinction matters: a soft 404 means the site has no ADF file, whereas "invalid" means the site attempted to create one but made structural errors.

Redirect Classification

When a file URL returns a 3xx redirect, the crawler follows up to 3 hops and classifies the redirect chain to determine whether the file genuinely exists at the destination:

This classification prevents false positives from servers that redirect all unknown URLs to the homepage or a custom error page instead of returning a 404 status code.

Readiness Scoring

Each domain receives an AI Readiness Tier from 0 to 5. The tier is computed from three inputs:

1. Valid ADF count — Number of ADF files with quality "minimal" or "complete" (files rated "invalid" do not count)
2. AI crawler classification — The domain's robots.txt policy toward AI user agents (see AI Crawler Analysis)
3. Schema.org presence — Whether the homepage contains structured data markup

The rules are evaluated in order — the first matching rule determines the tier:

Key design decisions in this model:

• Tier 5 requires explicit AI crawler permission — having no AI policy (no_ai_policy) qualifies for Tier 4 at most, because silence is not the same as consent.
• Tier 3 has two entry paths — a domain can reach Tier 3 either by having ADF files but blocking AI crawlers (a contradiction), or by having good structural foundations (Schema.org + open crawlers) but no ADF files yet.
• Schema.org is required for Tiers 4–5 — structured data demonstrates an understanding of machine-readable signals, not just ADF adoption.
• "Allows AI" includes both explicit allows and no policy — the absence of blocking directives in robots.txt means AI crawlers are permitted by default.

AI Crawler Analysis

We parse each domain's robots.txt and check for directives targeting 15 known AI user agents:

For each agent, the crawler checks whether robots.txt contains a matching User-agent: block and classifies the directives within it. Agents not explicitly mentioned inherit the wildcard (User-agent: *) behaviour. The crawler tracks whether each agent is effectively blocked — either by a direct Disallow: / or by wildcard inheritance.

Access Classification

Each domain is classified into one of five access policies based on the aggregate behaviour across all 15 agents:

1. Blocks All AI — All 15 AI agents are effectively blocked (directly or via wildcard inheritance).
2. Blocks Selectively — Some AI agents are blocked while others are allowed, or some are blocked but none are explicitly allowed.
3. Rate-Limits AI — AI agents are mentioned only with Crawl-delay directives, no allow/disallow rules.
4. Explicitly Allows — At least one AI agent has an explicit Allow directive, and none are blocked.
5. No AI Policy — No AI-specific directives in robots.txt (AI crawlers are permitted by default).

Comparison Standards

For context, we check for the presence of established web standards at their canonical locations:

This comparison contextualises ADF adoption relative to other standards that also require placing files at the domain root. It answers the question: how does ADF adoption compare with existing, well-established web conventions?

Schema.org Detection

Schema.org is checked separately by fetching the homepage and scanning the first 100KB of HTML for any of three markup formats:

1. JSON-LD — Presence of application/ld+json in the page source
2. Microdata — Presence of itemtype="https://schema.org/..." attributes
3. RDFa — Presence of vocab="https://schema.org/..." attributes

This is a presence check, not a quality assessment. We do not validate the correctness or depth of the structured data — only that the homepage includes at least one Schema.org reference in any supported format.

Known Limitations

• Sample scope — We crawl approximately 2,000 top domains. Adoption rates in the broader web (small businesses, long-tail sites, non-English markets) may differ significantly. Our data represents the largest, most visible websites.
• Domain-level only — We check the root domain only (e.g., example.com), not subdomains or subpaths. Organisations hosting ADF files on subdomains (e.g., docs.example.com/llms.txt) would be missed.
• Point-in-time snapshot — Each quarterly crawl captures the state at crawl time. Websites may add or remove files between crawls. Trends between quarters are indicative, not continuous.
• Soft 404 heuristics — Despite six detection methods, some edge cases may be misclassified. A site that serves a valid ai.txt file containing the phrase "page not found" in its content would be a false positive. We err on the side of caution (flagging as not found) rather than accepting ambiguous files.
• CDN and caching — Some domains serve cached or CDN-edge content that may differ from origin. The crawler sees whatever the CDN serves.
• Redirects — We follow up to 3 redirects. Domains with longer redirect chains are marked as errored. Our redirect classification uses heuristics (homepage catch-all detection) that may occasionally misclassify intentional redirects.
• Schema.org detection is shallow — We check for the presence of Schema.org markup in three formats (JSON-LD, Microdata, RDFa) but do not validate correctness, depth, or relevance. A page with a single, minimal Organization block is treated identically to one with comprehensive markup.
• Body truncation — File bodies are capped at 50KB and homepage bodies at 100KB. Exceptionally large files may be validated on a truncated version, though this is rare in practice for root-level text and JSON files.
• HTTPS only — The crawler only checks https:// URLs. Domains that are only available over HTTP (without TLS) are not covered.
• Specification evolution — The ADF specification may evolve between crawls. We validate against the specification version current at crawl time. Historical data reflects the rules in effect when it was collected.

Reproducibility

To support external verification, we publish the following:

• Domain lists — Sourced from the Tranco List (global) and UK domain rankings, frozen at the start of each crawl.
• Crawler identity — User-Agent: ADF-Adoption-Crawler/1.0
• Validation rules — Per-file required and recommended checks are documented in the ADF specification.
• Scoring model — The readiness tier calculation is fully described above, with no opaque weights or proprietary algorithms.
• Raw data — CSV downloads are available for every completed crawl on the research page.

Data Availability

All crawl data is available for download as CSV files on the research page, licensed under CC BY 4.0. Downloads include:

• Summary CSV — Per-domain results with readiness tier, valid ADF count, and AI crawler classification.
• Scores CSV — Detailed per-file quality tiers for every domain checked.
• Blocking CSV — Per-agent blocking data for every domain.

When citing this research, please reference the specific quarter (e.g., "Q1 2026") and link to the methodology page.

Crawl Schedule

Crawls run quarterly on the 1st of January, April, July, and October at 04:00 UTC. Batches are processed concurrently (5 domains at a time, 25 domains per batch) and results are aggregated automatically once all batches complete. The full crawl typically completes within 2–3 hours. Results are published within one week of the crawl completing.

The first crawl was conducted in February 2026.

Frequently Asked Questions

← Back to Research