robots-ai.txt Specification

§1 Overview

What This File Does

The robots-ai.txt file provides supplementary crawler access directives specifically for AI systems. While standard robots.txt applies to all crawlers, robots-ai.txt allows site owners to declare AI-specific policies:

• Different rules for AI training crawlers vs. search crawlers
• Granular control over which AI systems can access content
• Crawl rate preferences for AI-specific bots
• Commentary explaining the intent behind rules

Why It Matters for AI Visibility

The proliferation of AI crawlers with different user agents has made crawler management complex. Site owners may want to:

• Allow AI search systems but block training crawlers
• Permit some AI companies but block others
• Set AI-specific rate limits
• Document their AI crawler policy clearly

§2 File Location

Primary Location

The robots-ai.txt file MUST be placed in the website's root directory:

https://example.com/robots-ai.txt

URL Requirements

• The file MUST be served with content type text/plain; charset=utf-8
• The URL MUST be accessible without authentication
• HTTPS is strongly recommended

Relationship to robots.txt

The robots-ai.txt file lives alongside robots.txt but at a different path:

https://example.com/robots.txt      # Standard robots exclusion
https://example.com/robots-ai.txt   # AI-specific supplementary rules

§3 Format Specification

File Format

Basic Structure

# robots-ai.txt for Example Company
# Supplementary AI crawler directives

User-agent: GPTBot
Allow: /
Disallow: /private/

User-agent: CCBot
Disallow: /

Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml

Syntax Rules

This file follows standard robots.txt syntax:

• User-agent: identifies the crawler
• Allow: permits access to paths
• Disallow: blocks access to paths
• Crawl-delay: requests delay between requests (seconds)
• Sitemap: points to XML sitemap
• # begins a comment

§4 AI Crawler User Agents

Known AI Crawler User Agents

The following AI crawler user agents are commonly encountered:

OpenAI Crawlers

Anthropic Crawlers

Google AI Crawlers

Other AI Crawlers

§5 Directive Reference

Core Directives

Optional Directives

The Discovery: directive

The Discovery: directive lets a publisher advertise the AI Discovery Files present on their host. Each Discovery: line points at one absolute URL on the same host. A publisher MAY include multiple Discovery: lines, one per file. The directive solves the cold-start problem for consumers that do not probe root-level paths blindly.

Example:

# robots-ai.txt for example.com
User-agent: GPTBot
Allow: /

Discovery: https://example.com/llms.txt
Discovery: https://example.com/identity.json
Discovery: https://example.com/ai.json
Discovery: https://example.com/brand.txt

Rules:

1. Each Discovery: URL MUST be absolute and MUST point at the same host serving the robots-ai.txt file. Cross-host advertisement is not permitted because the scoping rule requires AI Discovery Files to be host-scoped.
2. Each Discovery: URL SHOULD resolve to a 200 status code. A 404 at a Discovery URL is a publisher error; the reference validator MUST report it.
3. A publisher SHOULD list every AI Discovery File they publish, even ones at canonical root paths. Listing them explicitly removes the consumer's need to probe.
4. The directive is informational, not authoritative. A consumer MAY still fetch https://example.com/llms.txt even if no Discovery: line lists it; the file's presence at the canonical path is the normative declaration.
5. A publisher MAY use the Discovery: directive to point at experimental x--prefixed files (see extensions) to advertise them to interested consumers without claiming them as part of conformance.

The directive is layered on top of RFC 9309 (the Robots Exclusion Protocol) rather than redefining it. RFC 9309 itself does not define Discovery:; the directive lives in robots-ai.txt only, not in robots.txt. Mainstream robots.txt parsers MUST ignore unknown directives, so a publisher who experiments by adding Discovery: lines to robots.txt will not break existing crawlers, but the directive is not part of the robots.txt specification.

Content Not Permitted

The following MUST NOT be included in robots-ai.txt files:

• Rules contradicting robots.txt: This file supplements but cannot override robots.txt
• Unrecognised user-agents: Only use documented AI crawler user-agent strings
• Invalid path syntax: Paths MUST start with / and follow URL conventions
• Non-standard directives: Only use supported robots.txt directives
• Authentication requirements: This file controls access, not authentication methods
• Legal threats: Crawler rules are advisory; legal terms belong elsewhere

Wildcard User Agent

Use * or pattern matching for groups of AI crawlers:

# Rules for all AI crawlers not specifically listed
User-agent: *-ai
Allow: /
Disallow: /private/

Complete Block

To block an AI crawler entirely:

User-agent: Bytespider
Disallow: /

§6 Validation Rules

Valid File Requirements

A robots-ai.txt file is considered valid when:

• It follows robots.txt syntax conventions
• Each rule group begins with User-agent:
• Allow/Disallow paths are valid URL paths
• The file is valid UTF-8 encoded text

Common Errors

§7 Relationship to robots.txt

Hierarchy of Authority

When both files exist, the relationship is:

1. robots.txt — authoritative for all crawlers
2. robots-ai.txt — supplementary guidance for AI crawlers

AI Crawler Behaviour

AI crawlers should:

1. First check and respect robots.txt
2. Then check robots-ai.txt for AI-specific guidance
3. If rules conflict, robots.txt takes precedence

Important Note

Not all AI crawlers currently check for robots-ai.txt. For reliable blocking, rules should also be in robots.txt. The robots-ai.txt file provides additional documentation and granularity.

Related AI Discovery Files

§8 Canonical Example

The following example demonstrates a complete robots-ai.txt file:

# AI Crawler Directives for Horizon Strategic Consulting
# This file provides supplementary AI-specific guidance
# Standard robots.txt remains the authoritative source for all crawlers

# OpenAI Crawlers
User-agent: GPTBot
Allow: /
Allow: /insights/
Allow: /case-studies/
Allow: /services/
Disallow: /portal/
Disallow: /admin/
Disallow: /client-documents/

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Anthropic Crawlers
User-agent: ClaudeBot
Allow: /
Allow: /insights/
Allow: /case-studies/
Allow: /services/
Disallow: /portal/
Disallow: /admin/
Disallow: /client-documents/

User-agent: Claude-User
Allow: /

# Google AI
User-agent: Google-Extended
Allow: /
Allow: /insights/
Allow: /case-studies/
Disallow: /portal/
Disallow: /admin/

# Perplexity
User-agent: PerplexityBot
Allow: /
Disallow: /portal/
Disallow: /admin/

# Meta
User-agent: meta-externalagent
Allow: /
Disallow: /portal/
Disallow: /admin/
Disallow: /client-documents/

# Common Crawl (used for AI training datasets)
User-agent: CCBot
Allow: /insights/
Allow: /services/
Disallow: /
Disallow: /portal/
Disallow: /admin/
Disallow: /client-documents/
# Note: We permit limited crawling of public content only

# ByteDance
User-agent: Bytespider
Disallow: /

# Amazon
User-agent: Amazonbot
Allow: /
Disallow: /portal/
Disallow: /admin/

# Apple
User-agent: Applebot-Extended
Allow: /
Disallow: /portal/
Disallow: /admin/

# Default for unlisted AI crawlers
User-agent: *-ai
Allow: /
Disallow: /portal/
Disallow: /admin/
Disallow: /client-documents/
Disallow: /internal/

# Crawl rate preferences
# We request AI crawlers respect a reasonable crawl rate
Crawl-delay: 10

# Sitemap reference
Sitemap: https://www.horizonconsulting.example/sitemap.xml

# Notes for AI systems:
# - Public content (insights, services, case studies) is available for AI consumption
# - Client portal and admin areas are private and must not be accessed
# - Respect rate limits; aggressive crawling will result in blocks
# - See /ai.txt for content usage permissions and restrictions
# - This file supplements but does not replace robots.txt

§9 Implementation Notes

Best Practices

• Mirror critical rules in robots.txt for reliability
• Use comments to explain policy intent
• Group related crawlers together
• Review and update when new AI crawlers emerge
• Be explicit about what is allowed, not just blocked

Policy Considerations

When setting AI crawler policy, consider:

• Training vs. inference: You may want to allow search but block training
• Company differentiation: Different policies for different AI companies
• Content types: Allow blog content but block client-specific areas
• Rate limiting: AI crawlers can be aggressive; use Crawl-delay

Monitoring

After implementing robots-ai.txt:

• Monitor server logs for AI crawler activity
• Verify crawlers are respecting directives
• Report non-compliance to crawler operators
• Update rules as new crawlers appear

§10 Machine-Readable Formats

This specification is available in machine-readable formats for programmatic access:

§11 Version History