Is Your Website Blocking AI Crawlers?
How to Check and Fix It

Why your website might be invisible to AI crawlers

Your website might be invisible to AI, and you'd never know it.

When someone asks ChatGPT, Claude, or Gemini about your industry, your competitors show up. You don't. Not because your content is weak or your site is new, but because something technical is preventing AI systems from accessing your information in the first place.

We can put a number on part of it. Our quarterly AI Discovery File adoption crawl checks the robots.txt of roughly two thousand websites every three months. In the Q3 2026 crawl, 9.9% of successfully crawled sites blocked GPTBot by name, and 13.7% blocked at least one AI crawler. That is only the blocking you can see in a text file. It does not count the sites whose firewall, CDN, or bot-protection rules quietly reject AI crawlers before robots.txt is ever consulted.

The difference between a website that AI systems can cite and one they ignore often comes down to a handful of configuration settings. A robots.txt rule that blocks GPTBot. A firewall that treats ClaudeBot as a threat. A JavaScript framework that serves an empty document to anything that isn't a browser. A missing llms.txt file that would have told AI systems exactly who you are and what you do.

This checklist walks through every technical barrier we see when running AI visibility checks across hundreds of websites. Each item is something you can verify and fix today. Where we have measured data, we have included it rather than guessing.

How AI crawlers reach your site

Before working through the checklist, it helps to understand how AI crawlers behave differently from search engine crawlers.

Traditional search crawlers like Googlebot visit your pages to build a search index. They follow links, read HTML, execute JavaScript, respect cache headers, and send referral traffic back when users click results. The exchange is simple: you let the crawler in, and you get search visibility in return.

AI crawlers work differently. Systems like GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot fetch content to train language models or to answer user queries directly. They usually read raw HTML without executing JavaScript. They look for specific files in your root directory. And the traffic ratio is dramatically skewed: where Google crawls roughly 5 times per referral it sends, Anthropic's crawl-to-referral ratio has been measured at over 38,000 to 1. AI crawlers sit inside a wider retrieval pipeline, and understanding that pipeline helps explain why a single blocked crawler can knock you out of citations entirely.

That line, "and that's real load," landed hard the first time I read it. We'd been thinking about AI visibility as a content problem: are your files in the right place, is the information accurate? But Prince is talking about something more visceral. Thousands of bots hammering your server for data they may never send a single visitor back for. When we started running AI visibility checks across client sites, we kept finding servers that were buckling under crawler traffic the site owners didn't even know existed. They were paying for hosting to serve human visitors, and half their bandwidth was going to machines. Cloudflare's 2026 threat report found that 94% of login attempts are now automated bots. That number puts the scale of non-human traffic into perspective.

Cloudflare estimates bot traffic will overtake human traffic by 2027. That's not a distant prediction. It's next year. The question isn't whether to engage with AI crawlers. It's whether you're engaging on your terms, or being excluded by default.

Four kinds of AI crawler, and why the difference matters

Treating "AI crawlers" as one category is the single most expensive mistake in this whole topic. There are four distinct jobs, and blocking one has nothing like the same consequence as blocking another:

That last row catches almost everyone out. Google never sends a crawler identifying as Google-Extended. It crawls with Googlebot and honours the Google-Extended token as a separate permission setting. Applebot-Extended works the same way. You cannot test for these two by watching your logs, because nothing will ever arrive under those names. They only exist as instructions.

Which websites are blocking GPTBot?

Most writing on this topic recycles the same two or three third-party statistics. We run our own crawl, so here is our own data.

Every quarter, the ADF adoption crawler fetches robots.txt from roughly two thousand websites and records which AI user agents are named, and whether they are allowed or disallowed. The Q3 2026 crawl completed on 1 July 2026 and successfully read 1,744 sites. Here is how often each AI crawler was blocked by name.

The full figures, in case you want to quote or check them:

The headline number most people expect is the one at the top: roughly one site in ten blocks the best-known AI crawlers. But the more interesting number is the one that never gets quoted. 84.2% of the sites we crawled had no AI crawler policy at all. No rule allowing them, no rule blocking them, nothing. Only 1.8% explicitly named AI crawlers in order to allow them.

That matters because it reframes the question. The web is not split between publishers who have thoughtfully opted out and publishers who have thoughtfully opted in. The overwhelming majority have simply never made the decision, and are getting whatever their CMS, their security plugin, and their CDN default happens to give them.

The pattern nobody mentions: sites block the trainer, not the visitor

Sort our data by crawler type and something consistent appears. Both OpenAI and Anthropic run a bulk crawler and a separate live-visit crawler. In both cases, the bulk crawler is blocked around twice as often.

Read one way, that is a coherent policy emerging across the web: don't take my content to train your model, but do come and fetch my page when a customer asks about me. It keeps the commercial upside of AI search while withholding the free training data. If that is what site owners intend, it is a defensible position and arguably the right one for most businesses.

Here is the honest caveat. Our data cannot tell the difference between a deliberate policy and a copied blocklist. Plenty of those rules will have arrived through a security plugin, a template, or a widely shared Gist that happens to name the famous training crawlers and omit the newer live-visit ones. The pattern in the chart is real. The intention behind it is inferred, and we can't prove it from robots.txt alone.

What we can say is that the outcome is better than a blanket block either way. If you take one thing from this section: check whether your rules distinguish between the two, because the accidental version of this policy works out considerably better than the accidental version of Disallow: /.

Reflex blocking has stalled. Deliberate allowing is rising.

We now have three quarters of comparable data, which is enough to see a direction of travel.

The share of sites blocking at least one AI crawler went 11.6%, then 13.5%, then 13.7%. The jump happened between Q1 and Q2. Between Q2 and Q3 it moved by two tenths of a percentage point, which is inside the noise you would expect from a partially different sample of sites.

Meanwhile the share explicitly allowing AI crawlers went 0.8%, 1.2%, 1.8%. Small numbers, but it more than doubled while blocking stood still. Sites that write an Allow rule naming an AI user agent are not doing it by accident. That is someone making a decision.

Two quarters is not a trend, and we will say so plainly when Q4 lands if it reverses. But the reading that best fits the data right now is that the reflexive blocking wave of early 2026 has run its course, and a slower, more deliberate sorting has started.

Does blocking AI crawlers hurt your citations?

This is the question people actually want answered, so here is a direct answer before the detail: yes, but only for some crawlers, and the ones most people block are not the expensive ones.

Work through it by crawler type, using the four categories from earlier.

• Block a search-index crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and you are removed from the index those systems search when composing an answer. You cannot be cited from an index you are not in. This is the block that costs you citations.
• Block a live-visit crawler (ChatGPT-User, Claude-User) and the AI cannot open your page for a specific person who is mid-conversation and asking about you. That is close to the highest-intent traffic on the internet, and you are refusing it at the door.
• Block a bulk training crawler (GPTBot, CCBot, Bytespider) and your content stays out of future model training. Your ability to be retrieved and cited live is largely unaffected, because that runs through the other two crawler types.
• Set a robots.txt switch (Google-Extended, Applebot-Extended) and you opt out of that vendor's AI training and grounding, with no effect on normal search indexing.

The trap is that the blanket rule catches all four at once:

# This blocks every crawler, including all four AI categories,
# and every search engine as well.
User-agent: *
Disallow: /

We have seen this in the wild on a business that had been trading online since the 1990s and could not work out why AI systems never mentioned it. The full story is in our write-up of a client blocking every crawler while submitting sitemaps to Google Search Console, which is a stranger combination than it sounds.

Now the limitation, because this deserves an honest answer rather than a confident one. We cannot show you a controlled study proving that unblocking a crawler causes citations to appear. Nobody can, at present. AI systems do not publish retrieval logs, citation behaviour varies between models and between weeks, and no publisher is going to run a randomised trial on their own visibility. What exists is the mechanism (you cannot be retrieved from an index you are excluded from) plus case-level evidence.

Our own case-level evidence: a removals company at full conformance was independently scored 92/100 by ChatGPT, and a three-week-old locksmith site reached the top of AI search results in its area. Both had every AI crawler allowed and complete AI Discovery Files. That is consistent with the mechanism. It is not proof of causation, and we are not going to dress it up as such.

Check 1: robots.txt and crawler access

This is where most unintentional blocking happens. Your robots.txt file and server-level rules determine which crawlers can reach your content.

Review your robots.txt for AI-specific rules

Open https://yourdomain.com/robots.txt in your browser. Look for rules targeting these user agents:

• GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI)
• ClaudeBot, Claude-SearchBot, Claude-User (Anthropic)
• PerplexityBot, Perplexity-User (Perplexity)
• Google-Extended (Gemini training and grounding switch)
• Applebot, Applebot-Extended (Apple Intelligence)
• CCBot (Common Crawl, used by many AI models)
• Meta-ExternalAgent, Bytespider (Meta AI, ByteDance)

If you see Disallow: / for any of these, your site is invisible to that AI system. This is a deliberate choice for some publishers, but many businesses have these rules without realising it, sometimes added by a CMS plugin or a security template. The 365i Robots.txt Checker can parse your file and test specific URL paths against your crawl rules to spot exactly what's blocked.

Paul Calvano's HTTP Archive analysis found that almost 21% of the top 1,000 websites now include rules for GPTBot, and that number grew from near zero to over 500,000 sites in under two years. Note that his 21% figure covers the top 1,000 sites, where publishers with licensing deals and legal teams are heavily over-represented. Our 9.9% covers a broader sample of ordinary business websites. Both numbers are right; they are measuring different populations, which is worth remembering whenever you see a single "X% of the web blocks AI" headline.

Calvano's data shows GPTBot going from zero robots.txt appearances to over 500,000 in just two years. That's not a measured response; it's a flinch. And I get it. When we first saw how aggressively some AI crawlers were hitting sites, the instinct to block everything felt rational. But sitting with it longer, and after watching businesses wonder why ChatGPT never mentions them, the cost of that reflex became obvious. They blocked the scraping, but they also blocked the citations, the recommendations, and the visibility. The smarter move, the one we keep coming back to, is to let the retrieval crawlers in and use AI Discovery Files to control what they learn. Give them the information on your terms rather than slamming the door.

Check for wildcard blocks

A common mistake is using a broad wildcard rule that catches more than intended:

# This blocks ALL bots, including AI crawlers
User-agent: *
Disallow: /

If your robots.txt starts with this, every AI crawler is locked out. You need explicit Allow rules for the crawlers you want, or a more targeted approach that blocks only specific paths.

Check your CDN and WAF rules

This is the most common source of unintentional blocking, and the one our robots.txt crawl cannot see. Content delivery networks and web application firewalls often classify AI crawlers as suspicious traffic. Cloudflare, Sucuri, Wordfence, and similar security tools may challenge or block requests from AI user agents by default. Cloudflare now blocks AI crawlers by default on new domains, which means a site can be blocking AI without a single line in its robots.txt.

Check your CDN dashboard for:

• Bot management rules that block "unrecognised" user agents
• Rate limiting thresholds that AI crawlers might exceed
• JavaScript challenge pages that block non-browser clients
• Country-based blocking that might exclude AI crawler IP ranges

If you're on managed hosting, your hosting provider should be able to whitelist known AI crawler user agents while keeping malicious bot protection active.

What a clean result looks like

Reading your own robots.txt tells you what you declared. It does not tell you what your server actually does when a request arrives claiming to be GPTBot. Those are different things, and the gap between them is where most silent blocking lives.

The free 365i AI Crawler Checker sends a live request as each of 14 AI crawlers and reports the response. Here is the result for this site, which is what a clean pass looks like:

Two rows in that grid are worth pausing on. Google-Extended and Applebot-Extended report "Not crawlable by name" rather than a status code, because no crawler ever arrives under those names. You cannot test them with a live request, only declare them in robots.txt. Any tool that claims to have successfully fetched your site as Google-Extended is reporting something that did not happen.

What to do when the block is at server level

Here is the situation this article was originally missing an answer for. You run the check, two crawlers come back rejected, and the rejection is nothing to do with robots.txt. It is a firewall or CDN rule inside infrastructure you may have no access to. On shared or managed hosting, you often cannot fix this yourself.

The useful move is to give your host something specific and technically precise, rather than asking whether they "block AI". The checker generates a message for exactly this, and it is worth reading even if you never run the tool, because it shows the shape of the request that gets results:

The third numbered request is the one that matters most, and it is the reason this message is phrased as a question rather than a complaint. A lot of bot protection is designed to reject unverified traffic claiming to be GPTBot while allowing the genuine crawler through from OpenAI's published IP ranges. That is correct behaviour, not a misconfiguration, and it produces exactly the same failed result in any user-agent-based test, including ours. Asking your host to confirm which of the two is happening saves everyone an argument.

Check 2: server and hosting configuration

Even when crawlers aren't blocked, poor server configuration can prevent them from reliably accessing your content.

Response times

AI crawlers are less patient than you might expect. If your server takes more than a few seconds to respond, crawlers move on. Google's own crawler ignores content that takes too long to fetch. AI crawlers behave similarly.

Test your server response time (not page load time, which includes client-side rendering). Your Time to First Byte (TTFB) should be under 500ms consistently. If it's regularly above 1 second, your hosting infrastructure needs attention.

For sites on shared hosting where noisy-neighbour effects cause intermittent slowdowns, moving to a dedicated WordPress hosting environment or managed cloud server with isolated resources can make a measurable difference to crawler reliability.

SSL/TLS configuration

AI crawlers require valid SSL certificates. Expired, self-signed, or misconfigured certificates will cause connection failures that happen silently. Your site looks fine in a browser (which displays a warning you can click past), but crawlers simply fail and move on.

Verify that your certificate is valid, covers all subdomains you use, and includes a proper certificate chain. Free certificates from Let's Encrypt work perfectly well; the important thing is that they're current and correctly installed. The 365i HTTPS Inspector scans for mixed content, insecure resources, and SSL/TLS configuration issues that might not be visible in a browser.

HTTP response codes

Your AI Discovery Files and key pages must return HTTP 200. This isn't theoretical: when we logged real AI crawler hits on a brand-new site, all 15 requests came back 200, which is exactly what you want to see. Common issues that break it:

• 301/302 redirect chains: A file at /llms.txt that redirects to /en/llms.txt that redirects to https://www.example.com/en/llms.txt. Each hop increases the chance a crawler gives up.
• 403 Forbidden: Server permissions blocking access to .txt or .json files in your root directory.
• 404 Not Found: The most basic failure. Files that simply don't exist yet.
• 503 Service Unavailable: Server under load or in maintenance mode when a crawler visits.

Content-Type headers

Your llms.txt should serve as text/plain. Your ai.json and identity.json should serve as application/json. If your server returns the wrong MIME type, parsers may reject the content even though it's valid.

Most web servers handle this correctly by default based on file extensions, but CMS platforms and custom routing can override defaults. You can test with curl -I https://yourdomain.com/llms.txt in your terminal, or use the 365i HTTP Header Inspector to check response headers, redirect chains, and status codes without leaving your browser.

Check 3: JavaScript and client-side rendering

This is the blocker people find hardest to believe, because the site looks perfect in a browser. Your pages can be fully open in robots.txt, fast, correctly certificated, and still deliver a blank document to every AI crawler that visits.

If your website relies on client-side JavaScript to render core content (React, Vue, or Angular single-page applications), AI crawlers will see an almost empty page. Googlebot executes JavaScript. GPTBot, ClaudeBot, and PerplexityBot typically don't. They read the raw HTML your server sends and work with whatever is in it.

That asymmetry is exactly why this goes unnoticed for so long. The site ranks in Google, so nobody suspects a rendering problem. Meanwhile every AI system sees a shell.

The two-minute test

You don't need a tool for this one. Fetch your own page the way a crawler does, with JavaScript disabled by definition, and look at what comes back:

# Fetch the raw HTML exactly as a non-rendering crawler receives it
curl -sL -A "GPTBot" https://yourdomain.com/ -o crawler-view.html

# How much text is actually in it?
wc -c crawler-view.html

# Is your main headline present in the source?
grep -o "Your Headline Text" crawler-view.html

If grep finds nothing, that text does not exist for an AI crawler. To read the result without counting bytes: open crawler-view.html in a browser. What you see is roughly what an AI system gets. If the page is empty, or shows only a navigation bar and a footer, you have found your problem.

A quicker version if you just want a signal: compare the byte count of the raw fetch against the rendered page size in your browser's developer tools. A content-rich page that returns 4KB of HTML is almost certainly rendering client-side.

How to fix it

Server-side rendering (SSR) or static site generation (SSG) ensures crawlers receive complete HTML. Every major framework supports one or both: Next.js and Remix for React, Nuxt for Vue, Angular Universal for Angular, and static generators like Astro, Eleventy or Hugo if the content doesn't need to be dynamic at all.

If a full rendering change isn't realistic in the short term, prerendering the most important pages for non-browser user agents is a legitimate interim step, and publishing AI Discovery Files gives AI systems a machine-readable description of your business that doesn't depend on rendering your pages at all. That is a mitigation rather than a fix, but it means an AI system knows who you are while you work on the underlying issue.

A category of commercial tools has appeared to detect rendering and crawler-access problems at scale, aimed mostly at large enterprise sites. For a single site, the curl test above tells you the same thing in about two minutes and costs nothing.

Check 4: login walls, paywalls and gated content

AI crawlers have no account, no password, and no session cookie. They get exactly what an anonymous visitor gets. If that's a login form, then that's your content as far as every AI system is concerned.

This one bites documentation sites hardest. Technical documentation is some of the most useful, most citable content on the web, and a great deal of it sits behind a login for reasons that made sense when the audience was human customers. The result is that AI systems answering questions about your product are working from your marketing pages and third-party blog posts, while your accurate, detailed, carefully maintained documentation is invisible to them.

The same applies to:

• Hard paywalls that serve nothing to anonymous visitors
• Cookie or consent walls that block content until a banner is accepted, since a crawler cannot click a button
• Age or region gates implemented as an interstitial
• Email-gated resources, including whitepapers and pricing pages behind a form
• Members-only knowledge bases and customer portals

How to test it

Open the page in a private or incognito window, logged out. If you can't read it, neither can an AI crawler. That's the whole test. For a stricter check, fetch it with curl as in the previous section, which won't carry any cookies at all.

What to do about it

Opening everything up is rarely the right answer, and we're not going to pretend the trade-off doesn't exist. Gated content is gated for commercial reasons. But there is a middle path, and it's the one we recommend to clients:

1. Publish a public subset. Getting-started guides, API references, and conceptual overviews are usually safe to open and are exactly what AI systems need in order to recommend you competently.
2. Describe what's behind the wall. If the material must stay gated, publish a public page that explains what it covers. AI systems can then tell someone that the answer exists and where to find it, which is far better than not knowing you have it.
3. Use AI Discovery Files as the machine-readable summary. This is precisely the job llms.txt and faq-ai.txt exist to do: state your services, scope, and answers in a form an AI can read without needing access to your gated pages.
4. Move consent walls out of the way of content. A consent banner does not need to prevent the page HTML from being served. If yours does, that's a fixable implementation choice rather than a legal requirement.

Check 5: AI Discovery Files

Even with crawler access sorted and server configuration right, AI systems still need structured information about your business. That's what AI Discovery Files provide.

These machine-readable files sit in your website's root directory and tell AI systems who you are, what you do, and how to represent you accurately. Without them, AI systems guess based on whatever they can scrape from your pages. With them, you control the narrative.

Start with llms.txt

The llms.txt specification defines the most widely adopted AI Discovery File. It's a plain text Markdown file that gives AI systems a structured overview of your business: name, description, services, exclusions, and contact information.

If you haven't created one yet, our step-by-step guide walks through the process. If you're unsure whether your business needs one, the short answer is: almost certainly yes.

Add supporting files

The full AI Discovery Files specification defines ten files, each serving a different purpose. After llms.txt, the highest-priority files are:

• identity.json (ADF-006): Structured business identity data in JSON format
• ai.txt (ADF-004): AI-specific permissions and preferences
• brand.txt (ADF-007): Official business name, naming rules, and terminology

You don't need all ten on day one. But the more files you publish, the harder it becomes for AI systems to misrepresent you. The quick start guide has a recommended implementation order.

Check for consistency

If your llms.txt says "Acme Web Solutions" but your identity.json says "Acme Digital Ltd" and your Schema.org markup says "ACME", AI systems can't confidently determine which name is correct. Inconsistency across files undermines the trust signal that AI Discovery Files are meant to provide.

Run your domain through the AI Visibility Checker to catch identity inconsistencies, missing files, and format errors in one scan.

Check 6: content structure and Schema.org

AI crawlers read your site differently from human visitors. They don't scroll, they don't watch videos, and as covered above, they usually don't run JavaScript. What they do is parse HTML and look for structured data.

Structured data and Schema.org

JSON-LD Schema.org markup gives AI systems machine-readable context about your content. At minimum, include Organization schema with your business name, URL, and contact information. Add Article schema to blog posts, LocalBusiness to location pages, and FAQPage to FAQ sections.

Schema.org and AI Discovery Files are complementary, not alternatives. Schema.org is embedded in your HTML and tied to individual pages. AI Discovery Files are standalone documents providing a complete business overview. You need both for full AI visibility.

Content locked inside images and canvas elements

Text inside an image is invisible to an AI crawler. So are the numbers inside a chart drawn on a <canvas> element, because canvas output is pixel data rather than text, even for crawlers that do render JavaScript.

This is a rule we hold ourselves to on this site. Every chart we publish, including the three in this article, is accompanied by the same figures in an HTML table. If a statistic only exists inside a picture, it may as well not be published as far as machine readers are concerned.

Meta robots and X-Robots-Tag

Check that your pages don't include <meta name="robots" content="noindex"> on pages you want AI systems to access. Also check for X-Robots-Tag HTTP headers that your server or CDN might add. These headers can block indexing globally without any visible indication in your HTML, which makes them one of the harder blockers to spot by eye.

Check 7: WordPress-specific blockers

WordPress powers over 40% of the web, and it introduces a few AI visibility issues specific to the platform.

Security plugins and bot blocking

Popular plugins like Wordfence, Sucuri Security, and iThemes Security include bot-blocking features that can reject AI crawlers. Check your security plugin settings for bot filtering rules and whitelist legitimate AI user agents.

Caching and robots.txt

Some WordPress caching plugins generate a robots.txt automatically or modify your existing one. Check that your live robots.txt (the one that crawlers actually see) matches what you expect. Plugins like Yoast SEO and Rank Math also manage robots.txt rules, and conflicting settings can create unexpected blocks.

AI Discovery Files on WordPress

The AI Discovery Files WordPress plugin generates and serves all ten AI Discovery Files directly from your dashboard. It handles the formatting, serves files at the correct URLs, and keeps everything consistent. If you're on WordPress, it's the simplest path to full AI visibility.

For WordPress sites that need more control over their hosting environment, bot management, and server configuration, 365i's WordPress hosting is built for exactly this kind of fine-grained control. And if you're migrating from a host that doesn't give you the access you need, free migrations make the switch painless.

How to tell if AI is actually crawling your site

Everything so far has been about removing barriers. This section answers the question underneath all of them: is my site being crawled by AI engines at all, and if so, which ones?

It is worth separating from the checks above. Those tell you whether an AI crawler could reach you. This tells you whether one actually did.

Read your server logs

The most direct evidence is your own access log. Every request records the user agent that made it, so AI crawler visits are already being recorded whether or not anyone has looked. On a Linux host:

# Every AI crawler hit in the current access log
grep -Ei "GPTBot|ClaudeBot|OAI-SearchBot|ChatGPT-User|Claude-User|PerplexityBot|Applebot|CCBot|Bytespider|meta-externalagent" access.log

# Count the hits per crawler, busiest first
grep -Eoi "GPTBot|ClaudeBot|OAI-SearchBot|ChatGPT-User|Claude-User|PerplexityBot|Applebot|CCBot|Bytespider|meta-externalagent" access.log \
  | sort | uniq -c | sort -rn

# Which of your AI Discovery Files are being read, and by what
grep -Ei "llms.txt|ai.json|identity.json|brand.txt|faq-ai.txt" access.log | grep -Ei "bot|gpt|claude|perplexity"

If those commands return nothing at all, that is itself a finding. Either nothing is reaching you, or your host is not retaining logs in a place you can read.

Use the plugin's crawler log

On WordPress, the free AI Discovery Files plugin logs every AI crawler that requests one of your discovery files and presents it without needing shell access. This is the AI Crawlers tab on one of our own sites, mcneece.com, over a seven-day window:

Three things worth pulling out of that screenshot, because they are easy to miss.

First, the most active crawler is ChatGPT-User, not GPTBot. That is the live-visit crawler: it fetched those files because a person asked ChatGPT something. Bulk training crawls are not the main event on a small site. Real questions from real people are.

Second, eight different bots across six operators turned up on an ordinary business site inside a week. This is not a phenomenon reserved for large publishers.

Third, the "0 bots blocked by robots.txt" card is the entire subject of this article rendered as a single number. That is what you are aiming for.

We have written up a longer version of this evidence, covering 11 days of logs and a Meta crawler sweep, in do AI crawlers actually read AI Discovery Files?.

The honest limitation of log-based checking

Logs undercount. A growing share of AI agent traffic arrives with an ordinary browser user agent string, either because the agent drives a real browser engine or because it deliberately presents as one. Those requests are indistinguishable from human visits in your access log.

So treat a log count as a floor, not a total. If you can see AI crawlers, you definitely are not blocking them. If you cannot see any, the safest interpretation is that you have a problem worth investigating rather than proof that nothing came.

If you actually do want to block AI

This article argues that most businesses block AI crawlers by accident and lose something they wanted. That is not the same as claiming nobody should ever block one. Some publishers have thought about it carefully and reached the opposite conclusion, and they are not wrong.

Blocking is a reasonable choice when:

• Your content is the product. Paid research, subscription journalism, and licensed datasets lose value if they are absorbed into a model that answers the question for free.
• You have a licensing deal, or want one. Several large publishers now sell access to AI companies. Giving it away undermines the negotiation.
• Crawler load is a real cost. On serverless or pay-per-request hosting, unmanaged crawler traffic shows up on the bill.
• You have a legal or contractual duty covering how client material may be processed.

If any of those apply, block deliberately rather than bluntly. That means naming the crawlers you actually want to exclude instead of reaching for Disallow: /, which also removes you from Google. The most common deliberate configuration keeps the retrieval crawlers and refuses the bulk ones:

# Keep the crawlers that let AI systems find and cite you...
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# ...and refuse the ones that collect content in bulk for training.
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Two caveats we would be doing you a disservice to leave out. robots.txt is a request, not an enforcement mechanism: well-behaved crawlers honour it, and badly behaved ones ignore it entirely. If you need enforcement, that is a firewall or WAF job, not a text file. And the robots-ai.txt specification offers finer-grained control by purpose rather than by user agent, which is a better fit when your policy is "training no, answering yes" rather than a list of names that goes stale every time a vendor ships a new crawler.

The point is not that blocking is wrong. It's that blocking should be a decision you made, not a default you inherited.

Test your AI visibility

Once you've worked through this checklist, verify everything in one place.

Scan your site for issues blocking AI crawlers

Two free scans cover different halves of the problem, and you want both.

The 365i AI Crawler Checker scans your site by sending a live request as each of 14 AI crawlers, so it catches server, WAF and CDN blocks that never appear in robots.txt. The AI Visibility Checker scans your domain across four dimensions: AI Discovery File presence, identity consistency, crawler access, and structural readiness. Each dimension is scored, and you get specific recommendations for anything that needs attention.

Both are free, take under a minute, and don't require an account. If you've made changes based on this checklist, run both to confirm everything is working.

Manual verification

For a quick manual check, test these URLs in your browser:

• https://yourdomain.com/robots.txt (should return 200, check for AI crawler rules)
• https://yourdomain.com/llms.txt (should return 200, text/plain)
• https://yourdomain.com/ai.txt (should return 200, text/plain)
• https://yourdomain.com/identity.json (should return 200, application/json)

If any of these return a 404, 403, or redirect, you've found something to fix.

List your site

Once your AI Discovery Files are in place and validated, submit your site to the AI Discovery Files Directory. The directory is monitored by AI crawlers and helps accelerate the discovery of your files. You can also browse the directory to see how other organisations have implemented their AI visibility infrastructure.

Frequently asked questions