Blocking Every Crawler in robots.txt? AI Can Never Recommend You

The email that prompted this post

Last week I delivered an AI Discovery Files pack to a client I like: a specialist electronics manufacturer whose domains have been registered since the early 1990s. Decades of organic results. Decades of paid search. A business that knows exactly who its customers are. And a robots.txt file, on their own physical servers, that reads like this:

User-agent: *
Disallow: /

Every crawler, blocked from every page. Not by accident, not by a WAF misfiring, but on purpose, as policy, for years. Their reasoning: the sites are hosted on their own hardware, and blocking crawlers protects the servers from overload. Instead of being crawled, they submit XML sitemaps to Google Search Console, three per domain, and consider the job done.

Here's what made me sit up. In the same email, the owner told me his main goal for 2026 is "increasing the odds of being in the list of companies that are contained in a typical Google AI search". He's frustrated that AI results surface companies that don't match what the user asked for. He wants the AI answers to be better. He wants to be in them.

He can't be. Not while that file exists. And he's not alone: I've now seen this exact configuration, deliberate, site-wide crawler blocking paired with dutiful sitemap submission, on several client sites. The owners all believe it's the smart, cautious choice. It's worth walking through exactly why it isn't, because the damage lands in two places, and the second one is permanent in a way the first isn't.

What Disallow: / actually does (and doesn't)

The first thing to understand is that robots.txt controls crawling, not indexing. Google's own documentation is blunt about it: a robots.txt file "is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google" (Google Search Central). The same page notes that a disallowed URL can still be indexed if other sites link to it.

So a site-wide Disallow doesn't cleanly remove you from Google. It does something stranger and worse. Google knows your URLs exist, because you handed them over in a sitemap, but it's forbidden from reading any of them. The result is what I think of as zombie listings: bare URLs in the index with no title Google chose, no description, no content, no understanding. Search Console dutifully files them under "Indexed, though blocked by robots.txt" and moves on.

That's the contradiction at the heart of this setup. A sitemap says "here are my pages, please come and look". A blanket Disallow says "you may not look". Submitting three sitemaps per domain while blocking the crawler that would read them isn't belt and braces. It's posting invitations to a party held behind a locked door.

My client's sites still show up for some queries because domains with thirty years of history and anchor text have residual signals. That residue is what "has worked out well as far as traditional search is concerned" actually means. It's a slowly draining battery, and it says nothing at all about the newer problem.

The bigger cost: AI can never recommend you

Losing search snippets is the visible damage. The invisible damage is bigger: an AI assistant can only recommend what its crawlers have been allowed to read. Block them all, and ChatGPT, Claude, Gemini, and Perplexity can never mention your brand, never cite your pages, and never put you in a shortlist. Not ranked lower. Absent. There is no residual battery here, no thirty years of goodwill to coast on, because AI answers are assembled from content the system can actually fetch and verify, a pipeline we walked through in how AI search actually works.

This is the part of the conversation where I feel the real urgency, because the traffic at stake isn't marginal. FirstPageSage's 2026 data shows ChatGPT-referred visitors converting above the industry baseline in all 32 sectors they measured, in some cases at nearly double the rate (FirstPageSage). These are customers arriving pre-sold, because an assistant they trust has already made the introduction. A blanket Disallow declines every one of those introductions, forever, silently.

"Imagine that you removed yourself from being searched by the world's most popular search engine. Needless to say, that's a bad decision, right?"

Adam Ipsen, Lead Content Strategist at Pluralsight, in Pluralsight's guidance on blocking OpenAI (verify quote at source)

Ipsen wrote that about companies rushing to block ChatGPT, and the line that follows it in his piece has stayed with me longer: pulling yourself off the grid leaves the space to be filled by something else, including misinformation. Reading my client's email, that's the sentence I kept coming back to. He complained that AI results "identify companies that do not match the users interest". Of course they do. The best-matching company told every AI crawler to go away. The assistant isn't being stupid; it's recommending the best of what it was permitted to read. He is, without meaning to, engineering the exact bad answers he's frustrated by.

And the Google-specific version of his goal is flatly impossible. Google's AI results are grounded in the ordinary search index that Googlebot builds. Block Googlebot and there's no content in the index for AI Mode or AI Overviews to ground on. Wanting Gemini to list your company while disallowing Googlebot isn't a long shot. It's asking to be quoted from a book you won't let anyone open.

The server overload myth

The stated reason for all this is server load, and I want to treat it fairly rather than mock it, because it was once true. If you were running a site on your own hardware behind a mid-1990s leased line, an aggressive spider really could hurt you. Domains registered in that era often carry policies fossilised from it. The fear outlived the facts.

The facts today: Googlebot manages its own crawl rate and slows down when your server signals stress with slow responses or errors, which Google documents in its crawl budget guidance. And the AI crawlers everyone worries about are a modest slice of traffic: Cloudflare's network measurements put AI bots, excluding Googlebot, at around 4.2% of HTML requests across 2025 (Search Engine Journal, reporting Cloudflare data). We've watched this first-hand too: when we logged every AI crawler hit on a new site for 11 days, the entire AI crawler load amounted to a few polite requests a day, each one answered in milliseconds.

If a particular bot ever does misbehave, the proportionate tools are all better than a blanket ban: rate-limit it at the CDN or firewall, block that one user agent, cache aggressively so crawler hits never touch your application. Blocking every crawler on Earth to protect a server is burning the shop down to save on heating.

Which bot controls what

Part of why owners reach for the blanket rule is that the per-bot controls look confusing. They aren't, once you see them in one table. This is the map I wish every robots.txt template site published:

Read the table and a legitimate middle path appears. If your concern is your content training someone else's model, block the training tokens (GPTBot, ClaudeBot, Google-Extended) and allow the search and retrieval tokens (Googlebot, OAI-SearchBot, PerplexityBot, Claude-User). You've protected the thing you wanted to protect and stayed recommendable. That's a defensible, deliberate policy. Disallow-everything is not a stricter version of it; it's a different decision with a different outcome, and I'd wager most owners running it never chose that outcome knowingly.

What sensible blocking looks like

None of this means robots.txt is useless. It's a good tool for the job it was built for: keeping crawlers out of the parts of a site that waste their time or expose things that don't belong in any index. Admin areas. Internal search result pages. Staging paths. Faceted URL sprawl. Blocking those is housekeeping, not hiding.

A sane robots.txt for a business that wants customers looks something like this:

# Keep crawlers out of the back office, welcome them everywhere else
User-agent: *
Disallow: /admin/
Disallow: /search/
Disallow: /staging/

# Opt out of AI training, stay in AI search
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

One more distinction saves a lot of grief. If you want a page kept out of results entirely, robots.txt is the wrong tool, because Google can't see a noindex instruction on a page it's forbidden to read. John Mueller, who has answered this question for site owners more times than anyone alive, puts it plainly:

"If these are pages that you don't want to have indexed, then using noindex would be better than using the disallow in robots.txt."

John Mueller, Search Advocate at Google, in a Google Search Central hangout, reported by iloveseo.com (verify quote at source)

What strikes me about Mueller's answer is how often it has to be given. The question he was answering, why do my disallowed pages still get traffic, is my client's setup viewed from the other end of the telescope. Both confusions come from the same root: treating robots.txt as an on-off switch for existing in search, when it's actually a door policy for reading. Once you hold that one idea, every rule in the file becomes a decision you can reason about, and the blanket Disallow stops looking cautious and starts looking like what it is: the single most self-destructive line you can publish in nine bytes.

And once the door is open, it's worth making what crawlers find unambiguous. That's the job of AI Discovery Files: an llms.txt that states who you are, a robots-ai.txt that declares your AI access policy in a form AI systems can read and cite, and the rest of the set keeping your identity consistent. Open the door first, then put up good signage. That's the whole discipline of AI Visibility Checking in one sentence, and it's why AI visibility matters more with every quarter of AI adoption.

Check what your robots.txt is actually doing

Here's the practical takeaway, and it applies even if you're certain your file is fine: most owners have never actually tested what their robots.txt does. It gets written once, inherited through rebuilds, appended to by plugins and developers, and quietly accumulates rules nobody remembers writing. The client in this story believed his setup was working well. So test it, don't trust it. It takes two minutes and both tools are free.

Then go one step further and test the thing robots.txt can't show you: whether AI crawlers can actually reach your site in practice. CDN rules and firewalls block AI crawlers silently even when robots.txt says welcome. The free AI Bot Checker from 365i runs live tests for 14 major AI crawlers against your homepage and reports which ones get through. And if you want the full picture, files, identity, consistency, and crawler access in one deterministic result, run the free AI Visibility Checker and see where you stand in the directory.

My client, to his credit, is still talking to me, and his AI Discovery Files pack is ready for the day the door opens. I hope he opens it. Thirty years of expertise deserves to be readable by the systems every buyer now asks first. What AI can't read, AI can't recommend, and no sitemap, no ad budget, and no legacy can route around that.

Frequently asked questions