Validation Framework

§1 Overview

What This Framework Tests

This framework tests whether AI systems are correctly interpreting the information declared in your AI Discovery Files. It verifies:

• AI can correctly identify your business
• AI accurately describes your services
• AI knows what you don't do
• AI respects your stated permissions

What This Framework Does NOT Test

This is not a tracking or monitoring tool. It does not measure:

• How often AI mentions your brand
• Your "ranking" in AI responses
• Sentiment or tone of AI responses
• Traffic from AI systems

Relationship to 365i Checker

The 365i AI Website Visibility Checker validates that your files exist, are correctly formatted, and are internally consistent. This validation framework tests the next step: whether AI systems are using that information correctly.

§2 Test Categories

The framework tests five dimensions of AI interpretation:

§3 Standard Test Prompts

Use these prompts with any AI system (ChatGPT, Claude, Gemini, Perplexity, etc.). Replace [BUSINESS] with your actual business name.

Identity Recognition (IR)

IR-01: Basic Identity

Expected: Correct business name, accurate business type, correct geographic scope (if applicable).

IR-02: Contact Verification

Expected: Contact methods match those declared in your llms.txt Contact section and identity.json.

IR-03: Name Recognition

Expected: If the alternate name is in your identity.json alternateNames or brand.txt, AI should confirm they're the same.

Service Accuracy (SA)

SA-01: Service Description

Expected: Services match your llms.txt Services section.

SA-02: Service Detail

Expected: Description aligns with how you describe this service in your AI Discovery Files.

Scope Boundaries (SB)

SB-01: Exclusion Recognition

Expected: AI correctly states this service is NOT provided (per your "What We Do Not Do" section).

SB-02: Geographic Scope

Expected: AI correctly identifies geographic limitations if you've declared them.

Citation Compliance (CC)

CC-01: Source Attribution

Expected: AI references your website or acknowledges its sources.

Permission Awareness (PA)

PA-01: Usage Permission

Expected: AI mentions restrictions if you've declared them in ai.txt or ai.json.

§4 Scoring Methodology

Score each test prompt response using this three-level rubric:

Evaluation Guidelines

Identity Recognition

• Business name spelled correctly?
• Business type accurately described?
• No conflation with other businesses?

Service Accuracy

• All major services mentioned?
• Descriptions substantially accurate?
• No services attributed that you don't offer?

Scope Boundaries

• Excluded services correctly identified as not offered?
• No false positives (saying you do something you don't)?

Score Interpretation

§5 Before/After Comparison

To measure the impact of implementing AI Discovery Files:

Baseline Assessment

1. Run all test prompts before implementing AI Discovery Files
2. Record responses and scores
3. Note specific inaccuracies or missing information

Post-Implementation Assessment

1. Wait 2–4 weeks for AI systems to potentially re-index
2. Run identical test prompts
3. Record responses and scores
4. Compare against baseline

Comparison Template

=== AI Visibility Validation Report ===

Website: [YOUR DOMAIN]
Baseline Date: [DATE]
Retest Date: [DATE]
AI System: [ChatGPT / Claude / Gemini / etc.]

BASELINE SCORES:
- Identity Recognition (IR): _/6
- Service Accuracy (SA): _/4
- Scope Boundaries (SB): _/4
- Citation Compliance (CC): _/2
- Permission Awareness (PA): _/2
- TOTAL: _/18

POST-IMPLEMENTATION SCORES:
- Identity Recognition (IR): _/6
- Service Accuracy (SA): _/4
- Scope Boundaries (SB): _/4
- Citation Compliance (CC): _/2
- Permission Awareness (PA): _/2
- TOTAL: _/18

CHANGE: +/- _ points

KEY FINDINGS:
- [What improved]
- [What remained unchanged]
- [Recommendations]

§6 Important Caveats

Key Limitations

1. Time Lag: AI systems may take weeks or months to incorporate updated AI Discovery Files into their responses. Don't expect immediate changes.
2. System Differences: Different AI systems consume files differently. A pass on ChatGPT does not guarantee a pass on Claude or Gemini.
3. Session Variability: The same prompt may produce different responses in different sessions. Run tests multiple times if results seem inconsistent.
4. Training Data vs. Live Files: AI systems may respond based on training data rather than your current files. This is especially true for well-known brands.
5. Not Deterministic: Unlike AI Visibility Checking (which validates file presence and format), these tests produce variable results. Use them for directional guidance, not absolute measurement.

Best Practices

• Run each test prompt 2–3 times to account for variability
• Test across multiple AI systems if possible
• Focus on trends over time rather than single-point measurements
• Use this framework alongside the 365i Checker for complete validation