Test Vectors

Canonical test suite for AI Discovery Files validators.

Status Draft

The test vectors themselves are under active development. This page documents the structure and location; the suite contents are being built in the ai-discovery-files spec repository. See specification conventions for status definitions.

Last updated: 11 May 2026

Purpose

The test vector suite is the canonical set of input-output pairs validator authors use to verify their implementations agree with the reference validator. Tests live in the public spec repository; this page documents where to find them, how they're structured, and how to apply them.

1. Where the tests live

The canonical home for test vectors is the ai-discovery-files repository on GitHub, under the test-vectors/ directory. The repository is public; tests are licensed under CC BY 4.0 alongside the specification text.

The tests are mirrored to this site at /specifications/test-vectors/ at deploy time. The mirror is a convenience for validator authors who want a stable URL; the spec repo is the source of truth and is updated first.

2. Test case structure

Each test case is a directory under test-vectors/<category>/<case-name>/ containing:

input/: The AI Discovery Files that constitute the test input. Typically a subset of the 10 ADFs (e.g. llms.txt, identity.json, ai.json) representing what a particular publisher might serve.
expected.json: The validator-output document the reference validator MUST produce for this input. Conforms to validator-output.schema.json. Stable fields (target, validator name held fixed, files[], contradictions[], conformance{}) MUST match; optional diagnostics MAY vary across validators.
readme.md: Plain-English description of what this case tests: what's interesting about the input, what the expected output demonstrates, which requirement IDs (e.g. ADF-IDENT-001) the case exercises.

3. Test categories

Tests are grouped into the following categories:

conformance/: Essential, Recommended, and Complete class verification cases.
contradictions/: cases where files disagree (business name mismatch, permissions vs robots.txt, llm.txt with independent content).
formats/: format-level edge cases (malformed JSON, BOM at start of file, unexpected whitespace, encoding issues).
http/: HTTP-protocol-level cases (301 chains, soft-404, 429 backoff, mixed content-type).
identity/: identity-resolution cases (multiple sources, alternateNames reconciliation, missing fields).
permissions/: permission-resolution cases (ai.json vs ai.txt, restriction precedence, robots.txt overrides).
versioning/: schema URL handling (versioned /v1/ vs unversioned, deprecated specification references).

4. Requirement identifiers

Each test case references one or more numbered requirement clauses of the form ADF-<FILE>-<NNN>. Example: ADF-IDENT-001 for "identity.json MUST contain a name field." The numbered clauses are stable across specification revisions; tests reference them so a validator failure can be traced to a specific normative requirement.

The full list of requirement IDs is published with each specification's data file under requirements[].id.

5. How to run the suite

For validator authors:

Clone the ai-discovery-files repository or fetch the mirror from this site.
For each test case, serve the contents of input/ as if they were files at the conventional paths on a real host (typically using a local fixture server).
Run your validator against the fixture host.
Compare your validator's output against expected.json. Required fields MUST match; optional diagnostics[] MAY differ but SHOULD cover the same semantic issues.

The reference validator (the AI Visibility Checker) runs the full suite as part of its CI pipeline. Conformant alternative validators SHOULD do the same.

6. Contributing tests

New tests are accepted as pull requests to the ai-discovery-files repository. A useful contribution:

Names a real-world publisher pattern (don't invent contrived cases when a representative real one exists)
Identifies the specific requirement IDs being tested
Includes input files, expected.json, and a readme.md explaining the case
Verifies the expected.json by running the reference validator before submission

See the governance page for the proposal lifecycle that applies to test contributions.

References

ai-discovery-files repository: canonical location for the test suite.
Processing Model: the algorithm the tests verify.
Validator Output Schema: the contract expected.json conforms to.
Conformance: the classes the conformance/ category exercises.
Governance and Editorial Process: how tests are proposed and accepted.
AI Visibility Checker: reference validator that runs the suite.