Your Next Website Visitor Might Be an AI Agent, Not a Person

The shift: AI agents are becoming website visitors

The morning we published this, Reuters reported that 73% of UK finance chiefs now expect AI to improve their company's performance, up from 59% at the end of 2025 and 39% two years ago. Deloitte surveyed 58 CFOs in the first two weeks of July. Boardroom conviction is climbing fast, and it is climbing for a specific reason: the conversation has moved on from which chatbot writes the best email to whether an AI system can reliably finish a real task.

Three days before that survey landed, OpenAI's CFO Sarah Friar published a scorecard for measuring exactly that. Her framework, "Useful Intelligence per Dollar", asks businesses to count completed outcomes: useful work finished, the full cost per successful task including human checking and rework, and whether each extra pound of AI spend buys more done-ness. Not tokens generated. Tasks completed. When the company selling the intelligence starts grading itself on finished work, you can take the shift from assistants to agents as official.

Google made the same move in plainer language. Announcing the I/O changes in May, Search VP Elizabeth Reid wrote that "we're entering the era of Search agents", with agents that research topics over time, ring businesses, help book local services, and act inside shopping journeys. We covered what those demos mean for business visibility at the time. The short version: an agent doing the choosing is a visitor doing the choosing, and it never sees your branding.

Before anyone panics, the counterweight: a July 2026 preprint from researchers at MIT FutureTech and Carnegie Mellon graded S&P 500 disclosures and found only 11% of firms had deeply integrated AI into business processes during 2025. Adoption talk runs ahead of adoption, as it always does. The tent goes up before the elephants arrive. But the direction is not in dispute, and websites are where a lot of it lands, because websites are where businesses keep their public facts.

Three kinds of machine visitors (and why they are not the same)

"AI traffic" gets discussed as if it were one thing. It is at least three things, with different intent and different value, and conflating them leads to bad decisions in both directions.

The volumes tell you why the distinction matters. DataDome counted 17.7 billion AI-associated requests in Q2 2026, up 45% on the previous quarter, but the same report shows Meta generating 9.1 billion of those requests while sending almost no referral visits, while ChatGPT produced 80 to 88% of measured AI referral traffic. Request count is not customer count. HUMAN Security's traffic study found the same shape from the other end: browser and agent traffic grew 7,851% across 2025, from a small base, yet still made up under 2% of AI traffic by December. Early, steep, and small. That is exactly the window in which preparation is cheap.

And the visits that do arrive are worth having. Adobe's analytics across US retail sites measured AI-referred traffic up 138% year on year in May 2026, with those visitors converting 54% better than average. These are people who asked an AI a question, got your business as the answer, and clicked through already half-decided. How your site ends up as that answer is the retrieval pipeline we walked through in how AI search actually works, and whether the crawlers that feed it can even reach you is a question with first-party server-log evidence behind it.

How AI agents read websites: the machine-readable representation

Here is the part most businesses have not internalised. An agent does not experience your website the way a person does. Google's developer guidance on building agent-friendly websites puts it more bluntly than we ever have:

"Agents don't look at your website on a monitor. They operate on a machine-readable representation of your site. The quality of this representation determines their performance."

Kasper Kulikowski and Omkar More, Google, in Build agent-friendly websites on web.dev (verify quote at source)

I have spent more than two decades judging websites by looking at them, and so has every client I have ever worked with. Design reviews, colour tweaks, hero image debates. Reading that paragraph in Google's own documentation was the moment the ground shifted for me: the visitor class that is growing fastest will never see any of it. It reads the accessibility tree, the raw HTML, the structured data, the machine-readable files. Everything we argue about in design meetings is invisible to it. The representation is the website now, and most businesses have never once looked at theirs.

Does the quality of that representation actually change outcomes? It has now been measured. A July 2026 study by Said Elnaffar and Farzad Rashidi ran 300 controlled shopping tasks against two versions of the same site, one conventional and one rebuilt to be machine-readable and predictable. The agent-ready version completed tasks at a strict 89.3% success rate. The conventional version managed 49.3%. Same business, same products, same agents. The difference was the representation. Google already scores a slice of this in Lighthouse, where this site passes Agentic Browsing at 3/3, and one of the three checks is an AI Discovery File.

AI Discovery Files: the reception desk your website is missing

A website without AI Discovery Files is a large business premises with no reception desk, no directory, no signs, and nobody to ask. A human visitor copes. They wander, open a few doors, and eventually work out what the company does. An AI agent arriving with a task and a time budget needs the building to explain itself: who the business is, what it offers, what it does not, which information is current, and where the important details live.

That is the whole job of the ten files. llms.txt is the front-desk summary of what the site is and where the key pages are. identity.json declares the business identity in structured form: legal name, locations, services, exclusions. ai.json carries machine-parseable business facts, and brand.txt settles naming and terminology so an agent describes you the way you describe yourself. An AI system does not read minds. If your site is vague or contradictory, the agent will still say something about you. It may just be wrong, or it may prefer the competitor whose site answered cleanly. The real danger was never invisibility alone. It is misrepresentation.

Do machines actually fetch these files? We stopped speculating and logged it. Our WordPress plugin records every AI bot request to the discovery files on mcneece.com, a deliberately low-profile personal site. In the rolling 30 days to 15 July 2026, 11 different bots from 8 operators, including OpenAI, Perplexity, Anthropic, Google, and ByteDance, read the files 43 times, with nothing blocked. The detail that surprised me most: brand.txt and faq-ai.txt were the most-read files at 8 reads each, ahead of llms.txt on 7, and OAI-SearchBot's single most-requested file was brand.txt. The naming file beat the headline file. Machines are already choosing which of your declared facts to read, on a site nobody promotes.

Why waiting for AI Discovery Files to "become a standard" is the wrong test

The most common objection we hear, from sensible people, is some version of: "I'll think about AI Discovery Files when they become a standard." It sounds prudent. It is the wrong test, twice over.

First, the web has never worked that way round. robots.txt ran the crawling behaviour of the entire web as an informal convention from 1994 and only became a formal IETF standard in 2022, twenty-eight years later. Nobody who waited for RFC 9309 gained anything by it. Adoption comes first; standardisation ratifies what already works. The same curve is visible now: our Q3 2026 crawl measured 9.4% of top websites publishing at least one AI Discovery File, up from 6.5% in Q1, with Cloudflare and Adobe joining the most AI-ready cohort, and Google now checks llms.txt inside its own Lighthouse tooling. The files are doing their job on live infrastructure today, whatever the paperwork says.

Second, and this matters more: the objection assumes the files are an SEO gamble that might pay off later. They are not an SEO tactic at all. They will not improve your Google ranking, and anyone selling them as a ranking trick is misleading you. Google said plainly that you do not need llms.txt to appear in AI Overviews, and we agreed with that narrow claim in detail. The files exist to help AI systems, and now AI agents, understand your company, your brand, and your website: what you offer, what you do not, how to cite you, and when. That job exists today, whether or not you have done it. If your next visitor is an AI agent rather than a person, you will wish you had thought about AI Discovery Files sooner.

Agentic commerce: the money rails are already live

If agents reading websites still feels theoretical, agents spending money should not. The payment infrastructure went live this year. Visa's Agentic Ready programme had more than 20 UK and European partners by April and is testing agent-initiated transactions with tokenised credentials and spending limits. Mastercard, Worldline, and ING completed a live end-to-end European agentic payment in production. Google introduced a commerce protocol for agents, merchants, and payment providers; OpenAI opened merchant product feeds into ChatGPT; Perplexity ships Instant Buy. McKinsey's scenario work puts agentic commerce at 3 to 5 trillion dollars globally by 2030. A forecast, not a fact, but the rails underneath it are facts.

Consumers are further along than the sceptics assume and further behind than the hype suggests, at the same time. NIQ measured 42% of US consumers using an AI tool to shop in the previous month, while only 5% had let an agent place an order autonomously. "We are witnessing the early stages of an industry-wide fundamental shift from search to decision," is how NIQ's North America president Liz Buchanan described it. The research half of the journey has already moved. The transaction half is following, with permission slips.

Merchants are not ready for either half. PayPal surveyed 498 US merchants this spring and found only about one in five had even 80% of their product catalogue structured and machine-readable. Adobe reached the same verdict from its own analytics: large parts of US retail websites are not entirely readable by machines, which limits how visible they are in AI results, and product pages are the weakest point of all. That is the page closest to the money. The CTO of WordPress VIP, Brian Alvey, drew the conclusion in a June interview:

"Companies that can serve both human and agent audiences will be the ones that survive."

Brian Alvey, Chief Technology Officer at WordPress VIP, in an interview with TechRadar Pro (verify quote at source)

My first instinct was to file "survive" under vendor drama. Then I thought about our own logs. A personal site we never promote is being read by eleven different AI bots a month, the payment networks have shipped the trust layer, and the merchants they surveyed are 80% illegible. Alvey also put it a way that has stuck with me since: people used to build websites for other people, and now you have to build them for the AI agents acting on behalf of those people too. That is not drama. That is a second audience arriving while most sites still only speak to the first one.

How to prepare your website for AI agents

The good news: preparing your website for AI agents is not a rebuild. It is three verifiable steps, and none of them requires a subscription.

Step one: open the door. None of the rest matters if agents cannot get in. Broad robots.txt rules, CDN bot protection, and firewall defaults quietly block AI crawlers and agents on a surprising share of sites; we met a business blocking every crawler it wanted to be recommended by this month. The free 365i AI Bot Checker tests 13 AI crawlers against your site with live requests and shows exactly who is blocked and where, and our technical checklist covers the fixes.

Step two: publish your machine-readable identity. This is where the reception desk gets built. Publish the AI Discovery Files at your domain root and Schema.org markup on your pages, so the machine-readable representation agents work from contains your declared facts instead of its best guess. The Quick Start guide gets the core files live in under an hour, and WordPress sites can generate and maintain all ten with our free AI Discovery Files plugin, which also logs which AI bots read them. If your site is built and managed for you, this is now a fair thing to expect from your developer: it is the standard Press Forge builds in by default, and hosts are starting to think about the same question from the infrastructure side, as 365i set out in WordPress AI agents and hosting readiness.

None of this is theoretical for the sites we work on. Three Press Forge builds went live with the full suite in place rather than bolted on afterwards, and you can fetch the files yourself right now. Cordley Solutions, a Herefordshire retail supplier selling established brands through UK marketplaces, publishes nine of the ten files, including an identity.json naming the registered company and a ai.json declaring what it supplies, backed by real product category and supplier pages. For a business whose customers increasingly arrive through marketplaces and comparison journeys, having a machine-readable answer to "who are these people and what do they actually supply" is not decoration. Lockerfella shipped its files on day one and topped ChatGPT and Gemini for its target term within weeks, and Brewood Removals carries all ten at Complete conformance. Different sectors, same decision: declare the facts before anything asks for them.

Step three: validate, then keep it true. Files that exist but contradict each other are worse than no files, because they hand the agent confident wrong answers. The free AI Visibility Checker validates all ten files, identity consistency, and crawler access deterministically: the same inputs produce the same score, with no prompt roulette. Then treat the published facts like the operational data they are. When prices, services, or opening hours change, the files change. A directory listing gives you a public, dated conformance record on top.

The honest limits of agent-ready websites

A claim this useful attracts overclaiming, so here are the limits, stated plainly. Autonomous agent purchasing is a minority behaviour today: 5% of consumers in NIQ's data, and Gartner found only 11% willing to delegate even lower-stakes purchase decisions. Agent traffic is under 2% of AI traffic by HUMAN's measurement. Most machine visitors this quarter are still crawlers, not customers.

AI Discovery Files will not rescue a site whose underlying facts are wrong. A declared price that disagrees with the page, a form that breaks, a service you quietly stopped offering: the files describe the business, they do not repair it. And none of this justifies paying a monthly fee to watch AI mentions of your brand bounce around, an argument we made at length in why AI visibility trackers are a waste of money. Fix the inputs you control. Ignore the outputs you cannot.

What remains after the caveats is the part worth acting on. The visitor mix on the web is changing for the first time since search engines arrived. The machines visiting now are early, countable, and easily served, and serving them costs a morning. Websites used to be built for people, then for people and search engines. Now there is a third reader at the door, sent by a customer, with a task to finish. Whether it understands your business when it arrives is already up to you.

Frequently asked questions