Cloudflare Radar put automated requests at 57.5 per cent of HTML traffic against 42.5 per cent human on 3 June 2026. The interesting part isn't the milestone; it's that the mechanisms meant to distinguish legitimate automation from everything else are either unenforceable or unfinished.
Why the old controls don't work
robots.txt is a request with no enforcement mechanism. A July 2026 measurement across 10,894 domains found that among pages cited by Google's AI Mode, 51.9 per cent were sites blocking at least one AI crawler in their robots.txt, versus 15 per cent across the full sample. Of sites specifically disallowing GPTBot, 39.5 per cent still served it a live response when it requested a page.
User-Agent is self-reported text. An operator can set it to whatever they want, including an ordinary browser string.
IP allowlists require maintaining a complete current list of every address a legitimate crawler might use, which breaks whenever infrastructure changes or a new operator appears.
What replaces it, and where it stops
Web Bot Auth: operator generates an Ed25519 key pair, publishes the public key as a JSON Web Key Set at /.well-known/http-message-signatures-directory, and signs every outgoing HTTP request with HTTP Message Signatures per RFC 9421. The site, or more realistically the CDN in front of it, verifies each signature against the published key. That genuinely ends user agent spoofing by anyone who signs.
Current state worth knowing before treating it as solved:
Googlebot's core indexing crawler doesn't sign. Only Google's newer AI browsing agent does, and Google still recommends IP verification and reverse DNS as a fallback.
The spec is a draft. The IETF working group was chartered in late 2025, with a best current practice operational document sent to the IESG in August 2026, so what major CDNs are running in production is unfinished by the standards body's own timeline.
Implementations have already diverged, including incompatible variants of the Signature-Agent header used for key discovery.
And the structural limit: a signature proves which operator an agent belongs to, not which human it's acting for. A site can confirm a request came from a given company's infrastructure without learning whether a person is waiting on a result or whether it's part of an unattended crawl at scale. Cloudflare is experimenting with RFC 7239's Forwarded header to carry an on-behalf-of signal, but their own documentation describes it inconsistently across pages, and it's a vendor proposal marked experimental rather than a standard.
This is the PGP lesson restated. A valid signature proves the private key was used. It tells you nothing about whether the signer's claims are true or whether you should do what they ask.
The bigger security problem, which identity doesn't touch
Browsing agents carry your active sessions and cookies as they navigate. Put instructions in a page in text a human wouldn't notice, and you can redirect what the agent does while it holds your authenticated access.
Indirect prompt injection is architectural rather than a bug. The model consumes your instructions and the page content as a single undifferentiated stream of text, with no structural marker separating trusted input from hostile input.
At Black Hat USA 2026, Brave security engineer Artem Chaikin presented "Attacking and Defending AI browsers", covering Comet, ChatGPT Atlas and Opera's AI browser. Every browser he analysed was vulnerable. Vendors including Perplexity and Anthropic shipped fixes for the specific findings, but as Dark Reading noted, there's no general fix short of not using browsing agents.
Where a VPN fits, which is narrower than you might want
A VPN changes what's observable at the network layer: your public IP, and what your ISP or local network can see. That's it.
Session access, prompt injection and device attestation schemes all operate at the application, browser or OS layer. Nothing a VPN does reaches them. Worth saying clearly since we sell one, and since "VPN protects you from X" gets applied to threats that don't live where a VPN operates.
The relevant second-order effect for anyone here: attestation systems can read a shared VPN exit address as an anonymising intermediary rather than a single trusted device, which means more CAPTCHAs, more failed checkouts and more login friction as these systems spread.
What can happen in the future
Verification keeps moving down the stack, from network signals to device and browser attestation, because the network layer has no signals left that can't be forged.
Anonymous connections don't get blocked; they get permanently rate-limited by friction. That's a worse outcome than blocking, because it never triggers a policy debate.
Prompt injection settles into the role phishing occupies: never solved, permanently mitigated, with a steady incident stream.
The web has had the underlying argument before. Web Environment Integrity in 2023 drew opposition from Mozilla over one entity deciding which browsers count as valid, with concrete concerns about Linux machines, older hardware and accessibility tools failing checks. Apple's Private Access Tokens raise a version of the same question through Secure Enclave attestation. Privacy Pass is the more interesting compromise, since blind signatures make the issuance and redemption moments cryptographically unlinkable, so a site learns a check was passed without learning who passed it.
Whether the next round lands closer to Privacy Pass or closer to Web Environment Integrity is a design choice, and it's being made right now while the spec is still in draft.
Full write-up: https://hide.me/en/blog/why-websites-are-trying-to-block-ai-crawlers/
https://i.redd.it/f13kttxb1anh1.png
Source: r/hidemeVPN · by /u/hidemevpn