Nine AI Bots Read Our Files in 30 Days.
Here's the Free WordPress Plugin That Logged Them.

Ask most WordPress site owners whether an AI system has ever read their website and you get a shrug. Not because they don't care. Because there is nothing to look at. Google gives you Search Console. Meta gives you an ads dashboard. ChatGPT gives you nothing at all.

So we instrumented one and watched.

What follows is 30 days of AI crawler logs from mcneece.com, a real WordPress site belonging to a one-person web design business in Northamptonshire. Not a demo. Not a mockup. Then the free plugin that captured it, what it publishes, and an honest account of what a log line does and does not prove.

What 30 days of AI crawler logs showed

Here is the headline panel for the 30 days to 4 August 2026.

• 55File reads
• 9Unique bots
• PerplexityBotMost active
• 0Blocked by robots.txt

And the full breakdown, bot by bot. Every one of these is a real request that hit the server and got a 200 back.

Three things in that table are worth more than the total.

First, Applebot is second. Ten reads from Apple, on a small business site in Kettering. Apple gets almost no attention in AI visibility conversations, which are dominated by ChatGPT and Claude, and it turns out to be one of the most diligent readers we have.

Second, the reading is spread across the whole set, not concentrated on llms.txt. PerplexityBot's most-read file is developer-ai.txt. GPTBot's is robots-ai.txt. If we had published only llms.txt, as most WordPress plugins do, we would have missed what those two bots actually came for.

Third, training crawlers and answer engines behave differently. ClaudeBot, Anthropic's bulk training crawler, called once in a month. Claude-User, which fetches pages when a person asks Claude something live, called seven times. Those are two very different relationships with your site, and lumping them together as "Anthropic" hides the useful one.

Now the same month, sliced by file rather than by bot.

A note on arithmetic, because you will spot it. The per-file column adds up to 58, and the summary tile says 55. The two panels apply the 30-day boundary slightly differently, so a handful of reads sit on one side in one view and the other side in the other. I am showing you both raw numbers rather than quietly reconciling them, because a dashboard that always adds up perfectly is usually a dashboard that has been tidied.

llms.txt leads, which is what you would expect from the only one of the ten with any name recognition. But it is 13 reads out of 58. The other 45 went to files that a single-file plugin would never have served.

The picture moves faster than you would expect

We published the same dashboard for the 30 days to 15 July 2026, three weeks earlier. That window recorded 43 reads from 11 distinct bots across 8 operators, and the most active crawler was OAI-SearchBot. Before that we published the first 11 days of raw logs from the same site, which was the point at which we stopped guessing whether anything read these files.

More reading, from fewer bots, led by someone else. That is the finding that makes the case for logging rather than checking. Had we looked once in July and called it done, we would have walked away believing OpenAI was our most attentive reader. Three weeks later that was no longer true.

This is a single small site, so treat the direction as interesting rather than settled. It is one data point, not a trend. But it is a real one, and it is more than most site owners have.

Why WordPress cannot normally tell you this

Your analytics package runs on JavaScript. AI crawlers do not execute JavaScript. So every AI bot that has ever read your site is invisible to Google Analytics, Fathom, Plausible and every other client-side tool, by design. They were never counted.

Raw server access logs do hold the answer, and if you have shell access you can dig it out. But most WordPress sites are on shared hosting where logs rotate every few days, arrive as gzipped archives, or simply are not exposed. Asking a small business owner to grep an access log is not a plan.

The result is a measurement gap that the security industry has been shouting about for a while.

"Invisible traffic is unmanaged traffic. And right now, most organizations cannot see this clearly enough to do anything meaningful about it."

Jérôme Segura, VP of Threat Research at DataDome, in DataDome Report Finds Most Organizations Flying Blind as Agentic Traffic Surges, March 2026 (verify quote at source)

Segura is talking about fraud and bot management, not AI visibility, and his company sells the thing that fixes it. I still can't get past how exactly it describes the smaller problem I keep running into with clients. When I show someone their crawler log for the first time, the reaction is almost never "that's a lot" or "that's not many". It's a pause, and then some version of I didn't know that was happening at all. They had no opinion about AI crawlers because they had no information to form one from. You cannot decide whether to allow, block, or court something you cannot see. Most site owners aren't making the wrong call about AI crawlers. They aren't making a call.

DataDome's own figures put the scale of it beyond argument: 7.9 billion AI agent requests across January and February 2026 alone.

The 10 AI Discovery Files the plugin publishes

Nearly every llms.txt plugin on WordPress.org stops at one file. The AI Discovery Files specification defines ten, grouped into three tiers so you can start small.

Look back at the crawler table and the argument for the full set makes itself. faq-ai.txt was the second most-read file on the site. brand.txt was third. Neither exists on a site running a plugin that only writes llms.txt.

The plugin generates all ten dynamically from your settings. Nothing is written to disk, so there are no stale files to clean up and no permissions to fight with, and every file updates the moment you change a setting.

The field most people skip, and shouldn't

The Identity tab has two boxes that look like afterthoughts and are the most valuable part of the whole exercise: Services NOT offered and Areas NOT covered.

An AI system asked "do they do X?" will reason from whatever it can find. If your site never says you don't do app development, a model that has seen a hundred web agencies who do may fill the gap for you. Writing the exclusion down closes it. On our own site those two boxes read:

Services NOT offered:
App Development
Hosting

Areas NOT covered:
Scotland
Northern Ireland

Four lines. They stop an AI recommending us for work we don't take and in places we don't serve, which is a customer's wasted enquiry and our wasted reply. We covered why this matters at length in the ten things your website must publish for AI agents.

AI crawler logging: which bot read which file

AI crawler tracking is the part with no real equivalent elsewhere, so it's worth being precise about what it does.

When a request comes in for one of the ten files, the plugin generates the response and then checks the user agent against a registry of 43 AI crawlers across 8 operator groups. On a match it writes one row: which bot, which file, when, and what status code. That is the whole mechanism.

What that buys you:

• Per-file attribution. Not "an AI bot visited" but "PerplexityBot read developer-ai.txt on 20 July". Because the plugin serves those URLs itself, it knows exactly which file was taken.
• It works behind a CDN. The plugin controls the discovery file responses, so edge caching doesn't blind it the way it blinds analytics.
• Off by default. Nothing is recorded, and no overhead exists, until you switch it on. You then tick exactly which crawlers to track.
• It prunes itself. A daily cron job deletes rows past your retention window, which defaults to 90 days and won't go below 30.
• A dashboard widget, so the number reaches you without going looking for it.

Now the honest part. The logger only sees requests to those ten files. It is not site-wide crawler analytics, and I would rather say so plainly than let you find out later. If GPTBot crawls two hundred of your blog posts and never touches llms.txt, this plugin records nothing. Plugins such as LLM Bot Tracker take the opposite approach and log AI bot hits across every request, and if whole-site coverage is what you're after, that is the right tool. They just don't publish anything for the bots to read.

The two jobs are different. One tells you who is walking past. The other puts something in the window and tells you who stopped.

There is a third question neither answers: who can get in at all. A crawler log only ever shows you the bots that arrived, so a bot blocked at your firewall leaves no trace in it, and its silence looks identical to disinterest. The free 365i AI Bot Checker closes that gap by sending a live request as each of the major AI crawlers and reporting which ones your server actually accepts. Run it once before you start reading logs, otherwise you can spend a fortnight wondering why GPTBot never calls when the answer is that it is being turned away at the door. Our guide to checking whether your website is blocking AI crawlers walks through the seven places a block can hide, most of which are nothing to do with robots.txt.

WordPress AI Connectors: filling the files without typing

The Identity tab has real depth: services with URLs, exclusions, service areas, operating hours, social profiles, company number, VAT number, jurisdiction. Filling that in by hand is twenty minutes of copying from your own website into a form, which is exactly the kind of task people abandon halfway.

WordPress 7.0 gave plugins a way to remove that chore, and it's worth understanding what actually shipped, because the version history gets muddled everywhere.

"WordPress 7.0 ships a built-in AI Client: a PHP API that lets plugins send prompts to AI providers and receive their results."

Felix Arntz, WordPress Core committer and founding member of the WordPress AI Team, in How to build an image generation plugin with the WordPress AI Client, May 2026 (verify quote at source)

What lands for me in that sentence is the word built-in, and I'll admit I underrated it when I first read the dev notes. I'd assumed this would go the way of every other "AI in WordPress" announcement: a bolted-on panel, a vendor lock-in, something to be endured. It isn't that. Core supplies the socket and stays out of the way, and the site owner decides who to plug in. I've been writing provider-specific integration code for years, the kind you rewrite every time an API shifts under you, and I did not expect the project to solve that quietly and properly at the plumbing level. It's the least glamorous thing WordPress shipped in 7.0 and probably the most consequential.

The precise history, since almost every article gets this wrong:

The plugin uses both layers. It registers seven abilities in an ai-discovery-files category, each with a permission callback and a JSON schema for its input and output:

add_action( 'wp_abilities_api_categories_init', array( __CLASS__, 'register_category' ) );
add_action( 'wp_abilities_api_init', array( __CLASS__, 'register_abilities' ) );

The seven are extract-services, extract-faq, extract-socials, extract-key-people, extract-taglines, extract-boilerplate and extract-brand-voice. Each reads a page you nominate on your own site and proposes entries for a field.

The prompts are deliberately tight. The services extractor is told, word for word, to "use the publisher's own service names verbatim" and "do not invent service names". The FAQ extractor is told to "use the publisher's exact wording wherever possible" and to return an empty array if the page has no clear question and answer structure. This is extraction, not creative writing, and that distinction is the whole point. Google's guidance on generative AI content is clear that automatically generated descriptions and structured data still have to be accurate and useful, and a model that paraphrases your services into marketing mush fails that test.

Nothing saves itself. Every extraction lands in a review modal where you tick what to keep. On our own site the Status tab records 15 imports run, each one reviewed.

Three caveats worth knowing before you go looking for the buttons:

• Core bundles no AI provider. WordPress 7.0 gives you the socket, not the electricity. You install a connector plugin for Anthropic, OpenAI or Google and pay that provider for what you use.
• Without a configured connector the buttons are disabled, with a hint reading "Install an AI connector and add your API key in Settings → Connectors to enable." Nothing breaks; the feature simply waits.
• Keys stored in the WordPress database are masked, not encrypted at rest. That is WordPress's own documented behaviour for Connectors, not a plugin quirk. On a shared host, weigh it.

And the whole feature is optional. Every one of the ten files works perfectly with the forms filled in by hand.

What it costs: free, GPL, no upsell

Free. GPLv2 or later. There is no pro version, no premium tier, no locked features, no licence key, no "upgrade to see your data" wall on the crawler log.

That deserves a word of scepticism, because "free" in the plugin directory often means "free until it matters". Here is what free actually covers: all ten files, the full crawler log with CSV export, validation, the dashboard widget, conflict detection and the AI import. The parts most plugins would gate are the parts we use most.

The one thing that can cost money is the optional AI import, and that money goes to your AI provider, not to us. Skip it and the plugin is free end to end.

Being straight about the flip side: this is a young plugin with a small install base, nothing like the millions behind Yoast or Rank Math. If you want the safety of a huge user base finding the bugs first, that is a fair reason to wait. It is actively maintained on WordPress.org, tested against WordPress 7.0.2, and the code is open, but I would rather tell you it is small than have you discover it.

Try the plugin without installing WordPress

You do not have to take any of this on trust, and you do not have to put anything on your live site to find out.

This link opens a complete throwaway WordPress install in your browser, with the plugin already active, on the plugin's own settings screen. Nothing is installed on your computer. Close the tab and it is gone.

Worth repeating that last point, because it is the one thing the demo gets wrong about the real experience. The Playground sandbox runs current WordPress, so the AI Connectors machinery is present, but there is no provider key in a throwaway site. The import buttons will be greyed out with a prompt pointing at Settings → Connectors. That is the plugin behaving correctly, not a broken demo.

How it compares to Yoast, Rank Math and other llms.txt plugins

Search for an llms.txt WordPress plugin and you get a long list, so it's worth saying where this one sits. If you already run a major SEO plugin you may have llms.txt covered: Yoast SEO added generation in version 25.3, and Rank Math and All in One SEO have their own. Credit where it's due, between them they put an llms.txt on millions of sites, which is more than any specialist plugin has managed.

Two claims you might expect me to make, and won't.

We are not the only plugin that logs AI crawlers. LLM Bot Tracker, LLMagnet, Odyssey and PufferSights all report AI bot activity in some form, at different scopes. Site-wide request monitoring and per-discovery-file attribution answer different questions, and describing all of them as "AI crawler analytics" without saying which would be sloppy.

We are not the only plugin using the new core AI APIs, either. Yoast registers three read-only abilities for its own scores; All in One SEO advertises Abilities and MCP operations. What I have not found elsewhere is a plugin using the WordPress AI Client to populate the AI-facing files themselves, which is a narrower claim than "first to use AI" and the only one I can support.

What is left after those subtractions is still a real gap: publishing the full set of files, and logging which bots take which one. Nothing else on the directory currently joins those two halves.

What the plugin does not do

Every section above is the case for it. This one is the case against, and if you only read one part of this article, read this one.

• It will not improve your Google rankings. Google states that llms.txt is not used by Google Search and neither helps nor harms ranking. Any plugin filed under generative engine optimisation or answer engine optimisation that promises rankings from a text file is selling you something.
• Most sites publishing these files see nothing read them. Ahrefs studied about 137,000 sites in May 2026: of the roughly 38,000 publishing llms.txt, 97% recorded no request for it in the study window. Our 55 reads are not typical, and I am not going to pretend they are.
• Nine of the ten files are a publisher-defined layer, not a vendor-endorsed standard. No major AI company has publicly committed to supporting ai.txt, identity.json, brand.txt and the rest by those exact filenames. Bots demonstrably fetch them, as the log shows. That is not the same as an endorsement.
• A fetch is not a citation. A 200 response proves delivery. It does not prove the content was indexed, retained, used in an answer, or attributed to you.
• User agents can be forged. DataDome found PerplexityBot the most impersonated AI agent, with about 2.4% of requests bearing its name classified as fraudulent, and 16.4 million spoofed meta-externalagent requests. Our top-ranked bot is the most-faked one on the web, which is worth sitting with.
• Caching can hide requests. If a physical file or an edge rule serves a discovery file without invoking PHP, the plugin never sees it and your log undercounts.
• Logging IPs and request histories carries obligations. Under UK GDPR that is personal data with retention and minimisation duties. Set a sensible window and mention it in your privacy policy.

None of that makes the files pointless. It makes the honest claim a narrow one: these files let you state, in a form a machine can parse, who you are and what you do not do, and the log tells you whether anything came to collect it. That is worth fifteen minutes and nothing at all. It is not a ranking strategy, and the gap between those two sentences is where most of this industry's nonsense lives. We argued that case in full in why AI visibility trackers are a waste of money.

How to set up AI Discovery Files on WordPress

If you came here looking for how to add llms.txt to WordPress, this is the short answer, and you get the other nine files with it. About fifteen minutes, most of it typing your own business details.

1. Install the plugin. Plugins, Add New, search "AI Discovery Files", activate. Needs WordPress 7.0 and PHP 7.4 or later.
2. Fill in Identity. Name, contact, location, services with links. Then the two boxes people skip: services you don't offer, areas you don't cover.
3. Set Permissions. How AI may use your content (allow with attribution is the recommended setting), whether training is permitted, your licence, your preferred citation format, and which crawlers may access the site.
4. Enable files on Status. Start with the Essential pair, add Recommended, then Complete. Hit Validate All.
5. Turn on crawler logging. Crawlers tab, enable, choose a retention period, tick the bots you care about.
6. Check it's live. Visit yourdomain.com/llms.txt. If you have a physical robots.txt in your webroot, WordPress can't edit it, so copy the plugin's block in by hand.
7. Leave it two weeks, then read the log. Crawlers move slowly. A week is too short to tell you anything.
8. Re-run the AI Visibility Checker. It scores the finished result from the outside, the way an AI system sees it, and catches contradictions between your files that the plugin's own validator will not spot.

The robots.txt step catches people out, so here is the block. Every line is a comment, which makes it safe for any parser while remaining readable by tools that go looking:

# AI Discovery Files
# Specification: https://www.ai-visibility.org.uk/specifications/
# llms.txt: https://example.com/llms.txt
# ai.txt: https://example.com/ai.txt
# ai.json: https://example.com/ai.json
# identity.json: https://example.com/identity.json
# brand.txt: https://example.com/brand.txt
# faq-ai.txt: https://example.com/faq-ai.txt

If you would rather understand the file before a plugin writes it for you, our step-by-step guide to writing llms.txt by hand covers the format and the sections that matter.

Not on WordPress? The Quick Start guide walks through the same files by hand, and the examples page has a working file of each type to copy. Once you're live, add your site to the directory so the files are independently verified and listed. Sites that would rather not do any of it themselves can have 365i handle it, and 365i Web Design builds it into new sites as standard.

Frequently asked questions