A Rounding Error

Over twenty-four hours, the single largest AI crawler on one consultant's website arrived roughly 1,500 times under the name of a nonprofit research archive, sent back nothing, and asked for his SSH keys. Then his AWS config. Then his Firebase service account key. Across about a hundred such paths it made 1,028 requests and took 6.7 megabytes, with zero referrals, and the number of requests for anything he had actually written rounds to nothing.

It never appeared in his security log. Security logs record requests that trip a rule, and he was not blocking it, so it was served and left no mark. It appeared in exactly one place on his entire dashboard: the AI crawler view, sitting in a list beside ChatGPT-User and Googlebot, credited to a research nonprofit and counted toward his AI crawler totals.

That is the whole problem in one request. Cloudflare's CFO told analysts that machine traffic could reach a thousand times human traffic within five years, and that is the number everyone will quote. The number that should actually worry you is the one above: the same request was an audience to one of that company's systems and invisible to another, at the same moment, on the same server.

The measurement is real, and the crossover already happened

Take the projection seriously first, because it is better sourced than most things in this category. On Cloudflare's second-quarter earnings call, CFO Thomas Seifert told analysts that "if the current trends continue, we think in five years, non-human traffic will be as much as 1,000 times as much as human traffic," and then delivered the line that will outlive the call: "humans will be a rounding error on the internet, not because human traffic goes down, but that's just how fast we're seeing non-human traffic grow."

He also volunteered, unprompted, "with the big caveat that I have called it wrong at every point along the way." Credit for that. But notice the direction of the errors. Cloudflare had expected machine traffic to overtake human traffic in 2027. It happened in May 2026. He has been wrong by being low, which is the strongest available argument for not discounting him.

Underneath the forecast is a measurement, and the measurement is the part that matters: Cloudflare's own writing the same week put it plainly, that fewer than half of all HTML page requests now come from a human.

Here is the operational consequence, and it is duller and more useful than the headline. Your analytics platform is a report on the minority. GA4 filters bots out. That is not a defect, it is the entire purpose of the thing, and it does that job well. But the effect is that the majority of what arrives at your infrastructure is deliberately absent from the dashboard your marketing team reads and fully present on the invoice your finance team pays. Nobody designed that gap. It is just what happens when a measurement tool built for an audience of people meets an audience that is mostly not people. We wrote about the cost side of this in FinOps at Render Scale; this is the same argument arriving from the other end of the wire.

A file that says please

The control everyone believes they have over this is robots.txt, and it is worth being blunt about what that file is. It is a text document on your own server containing a polite request, which the party being governed decides whether to honour.

New data puts numbers on how that is going. Matt G. Southern at Search Engine Journal, working from TollBit's State of the Bots report for the first half of 2026, reports that across the European sites in the study about 15% of identified AI page-fetchers reached URLs the sites had marked disallowed. It concentrates in a few agents: ChatGPT-User, Bytespider and Youbot each reached disallowed pages on nearly half of the European sites that had explicitly listed them, and ChatGPT-User reached the most sites of any of them.

OpenAI's own documentation explains why, and it is not a bug report. ChatGPT-User fetches a page when a ChatGPT user asks a question, and because the action was initiated by a person, OpenAI says robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores the file for the same reason. Anthropic takes the other position and states that all three of its bots respect it. TollBit, sensibly, counts any request to a disallowed URL as a bypass regardless of what the operator claims.

Sit with the fact that this is a methodology decision at all. Whether your control worked is currently a matter of interpretation between you and the party you were trying to control. Which is the identical argument we made on Saturday in Seven Days, By Default about a vendor's safeguards: a disallow line is a setting, not a constraint, and the difference is whether the party bound by it got to choose.

There is also a trap in the obvious defensive move, and it costs money. According to OpenAI's documentation the agent that decides whether you appear in ChatGPT's search results is OAI-SearchBot, not ChatGPT-User. As Southern puts it, sites that block both to keep AI traffic out "have traded away the visibility half of that deal and kept a fetching control that carries a carve-out." You lose the referrals and keep the fetches. That is the worst of the available outcomes and it is what you get by default if you treat every AI user agent as one thing.

His single best sentence is the one to write on the wall: "Server logs or CDN records show what actually arrived. The file only shows what you asked for."

And the market is already correcting in the way these things always correct, which is that the honour system gets replaced by enforcement. Cloudflare is moving the decision to the network layer. From 15 September, new domains added to Cloudflare will have Training and Agent crawlers blocked by default on pages carrying ads, while Search crawlers stay allowed. Compliance stops being the crawler's decision.

Your monitoring classified an attack as an audience

Now the part that is genuinely new, and it comes from somebody looking at his own logs rather than at a forecast. Slobodan Manic pulled Cloudflare's AI crawler view for his own site for the twenty-four hours ending the evening of 7 August.

About 3,000 requests, roughly a third of them unsuccessful, a failure figure up more than 1,000% on the previous period. By crawler: CCBot 1,510, ChatGPT-User 375, ClaudeBot 296, Googlebot 245, PetalBot 107, and thirteen others sharing 353 between them.

CCBot is Common Crawl's crawler, the long-running nonprofit web archive whose corpus trained a good share of the models everybody now argues about. Being his largest visitor is, on paper, unremarkable. Then he read the paths.

/id_rsa. /id_ecdsa. /private-key. /ssl/localhost.key. /key.json. /serviceAccountKey.json. /.aws/config. /actuator/configprops. /api/v1/env. /Dockerfile. /values.yaml. And /@fs/proc/self/environ, which is an attempt at a known path-traversal bug in a development server. The closest that traffic came to his actual writing was a request for /blog/wp-login.php, aimed at a website that has never run WordPress.

As he says himself, whatever that is, it is not reading his pages before it asks for things. It is working through a list, the same list it works through everywhere, and his website is a row in a loop.

One caveat, which he raises himself and we are carrying because it matters. He could not verify the source addresses and therefore could not prove impersonation, because per-request IP data was not available on his plan. Common Crawl publishes the test: genuine CCBot traffic comes from documented address blocks and reverse-resolves to hostnames ending in crawl.commoncrawl.org. So what is established here is what arrived, what it asked for, and how it was labelled. Not who sent it. Anyone with the full logs could settle the rest in a minute.

What is not in doubt is the classification failure, and it generalises to every company reading this. Three systems saw that day. Analytics did not report it, because it filters non-human traffic by design. Security did not report it, because security logs record rule trips and nothing was blocking it. The AI crawler dashboard reported it, and called it an audience. Not one of those systems malfunctioned. Each did exactly the job it was built for, and a credential sweep fell cleanly through the seam between them.

We have made this argument before in a different industry. In The Bucket Called Other the point was that a wrong taxonomy fails silently, because the report still renders. Here the report not only renders, it renders the attack as growth.

The wordlist already knows about your agents

Two paths in that log are new, and they are the reason we moved this piece up the queue.

/.mcp.json, requested 30 times. /.continue/config.json, requested 24.

Those are agent tooling configuration files: an MCP server definition and a coding assistant's settings. Both routinely contain API keys and access tokens, for the straightforward reason that keys and tokens are what you put in them so an agent can reach your services.

Somebody has added agent credentials to the standard secret-scanning wordlist. No announcement, no disclosure, no advisory. The same automated sweep that has been asking every website on the internet for /.env since roughly forever now also asks for the file listing which tools your agents can call and what they authenticate with. Manic's summary is the one to act on: the wordlist arrived before most people finished writing their first MCP server.

This is the third leg of an argument we have been building all month. In Ask For The Prompt we argued that MCP replaced the auditable data pipe with an instruction surface you have never read. In Data Never Leaves we argued that internally generated apps get a badge and a production database with nobody owning app 212. Here is the part neither piece anticipated: the configuration that binds those two together is now being swept for at internet scale, by automation that has never heard of you and is not targeting you.

If your organisation has adopted any agent tooling in the last year, somebody is already asking your webserver where you keep its credentials. The useful question is not whether they are asking. It is whether you would find out if the answer were ever anything other than a 404.

The company owns the meter and the valve

Manic makes an observation about his own primary source that we would want somebody to make about ours, so we are carrying it in full rather than quietly using the numbers.

In the first week of August alone, Cloudflare produced a bot-traffic projection on an earnings call, a blog post quantifying how much of the web is no longer human, an agent-readiness scanner to tell you that you are not ready, an AI-visibility product to score you, a bridge to expose your website's tools to agents, and a default that starts blocking some of those agents in September unless you decide otherwise. In his words: "The company measuring the problem, framing the problem, and selling the fix is one company, and they now own both the meter and the valve."

He is careful about the other side of it, and so are we. He has publicly backed pay-per-crawl and the principle that site owners should get a real choice over which machines get in. Every one of those products is a reasonable response to something real, which is exactly what makes the pattern worth noticing rather than dismissing. His posture is the right one and we will borrow it directly: take the traffic numbers seriously, and take the framing with the salt it deserves.

It is also worth crediting Cloudflare with publishing its own counterweight the same week. Its agentic-internet writing notes that a great deal of traffic from well-behaved bots is re-fetching pages that have not changed, running to billions of requests, which it describes as "an enormous amount of machine effort, attached to no outcome at all."

Machine effort and machine demand are different quantities. That is the distinction to carry into every conversation you have about this for the next two years. A rising line on a traffic graph does not tell you whether you have acquired an audience or acquired a bill.

The playbook

Six moves. None of them require buying anything, and the first one takes an afternoon.

1. Read the paths, not the totals. Open your CDN's crawler analytics for one ordinary day and sort by requested path rather than by volume. You are looking for three things: requests for content you made, requests re-fetching what the crawler already has, and requests for files you never published. Those are three different businesses arriving on the same chart.

2. Reconcile your three systems. Pick that same day and count the same requests in analytics, in security, and at the CDN. Write down why the numbers differ. If nobody in your organisation can explain the gap in a paragraph, the gap is where your next incident lives.

3. Stop treating robots.txt as a control. Keep the file, write it properly, and then verify at the log or network layer what actually arrived. The file records your preference. Your logs record events.

4. Do not block your way out of visibility. Before you disallow anything, separate the agent that fetches pages from the agent that decides whether you appear in an assistant's answers. They are different user agents with different consequences, and blanket-blocking buys you the loss and not the protection.

5. Put the agent config files on your secret-scanning list today. /.mcp.json, /.continue/config.json, and whatever your own stack's equivalents are. Then check your logs and find out whether any of them has ever been served rather than refused.

6. Price the machine audience before somebody prices it for you. Bandwidth, origin CPU, cache-miss rate, per crawler, per month. September is when at least one platform starts making that decision on your behalf by default, and a default you did not choose is a decision you did not make.

Our position

In July we wrote Your Website Is an API Now, which argued that your site had stopped being a page for people and become a data source for machines. Four weeks later, this is the operations bill for that argument arriving, and it is larger than we expected.

The thing worth taking away is not the ratio. It is that the machine audience is simultaneously a marketing problem, a security problem and an infrastructure-cost problem, which in most organisations means it is nobody's problem, and that is precisely how a credential sweep spends a day taking megabytes off a website while being counted as a reader. Every system involved was working correctly. The failure was that no one person was accountable for the whole request.

Seifert's line is a good headline and a poor plan. "Humans will be a rounding error" is a statement about volume, and volume is the one part of this you cannot do anything about. The question actually in front of you is smaller and answerable: for one ordinary day on your own domain, how much of what arrived wanted something you made, how much was re-fetching what it already had, and how much was looking for your keys?

Almost no company can answer that today. It is an afternoon of work. And the graph is going to keep climbing either way, so the only thing you get to decide is whether that line represents an audience or an invoice. You will not be able to tell from the totals.

Sources

Every claim above is carried from the reporting below, credited in the body and linked here. Where a figure reaches us second-hand we have said so, and where the reporter could not verify something himself, neither have we.

• Slobodan Manic, "Cloudflare: Machine Traffic Could Hit 1,000x Human Traffic In 5 Years", Search Engine Journal, 14 August 2026. Source for the entire crawler log analysis, the per-crawler counts, the requested paths, the security-log blind spot, the two agent config paths, the Cloudflare product pattern and the "meter and the valve" quotation. Thomas Seifert's earnings-call remarks and Cloudflare's own engineering writing reach us through this piece rather than directly; we have not read the call transcript. Manic states that he could not verify the source addresses of the CCBot-labelled traffic and therefore could not prove impersonation, and that caveat is carried in the body.
• Matt G. Southern, "OpenAI Says Robots.txt May Not Apply To ChatGPT's Fetch Bot", Search Engine Journal, 14 August 2026. Source for the TollBit State of the Bots findings, the disallow-bypass rates by agent, OpenAI's and Perplexity's documented positions, Anthropic's contrary position, the OAI-SearchBot distinction and the 15 September Cloudflare default.
• TollBit, State of the Bots, first half of 2026. The underlying dataset for the bypass figures, reported here as Southern reports it rather than read directly.