Vincent Schmalbach published a great piece of detective work yesterday: TIME Is Serving AI Bots a Different Website, With Ads Built In. Fetch a TIME article as Chrome and you get 303 KB of HTML. Fetch it as ClaudeBot or PerplexityBot and you get 13 KB of clean markdown at the same URL — with sponsored FAQ blocks inside that no human reader will ever see, and a fresh ad-impression UUID on every request. TIME says bot traffic already outnumbers human traffic on most days.
My first reaction wasn’t shock. It was: hang on, I already built most of that.
The non-evil version was already running here
Every post here has a raw-markdown twin published next to the HTML — the header image of this post is the twin of my wasm-not-docker post, exactly as ClaudeBot receives it. Add llms.txt, a <link rel="alternate" type="text/markdown"> on every post, and a robots.txt that welcomes the AI crawlers instead of fighting them. A Jekyll plugin generates it all at build time.
But TIME’s experiment exposed my one gap: discovery. My twins lived at .md URLs a bot had to know about. Most crawlers just fetch the canonical URL and take what comes back. The negotiation has to happen at the canonical URL. Fifteen lines of Apache fix it:
RewriteCond %{HTTP_USER_AGENT} (GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|CCBot|Amazonbot|...) [NC]
RewriteCond %{DOCUMENT_ROOT}/$1.md -f
RewriteRule ^(.+)\.html$ /$1.md [L]
Known AI crawlers now get the twin at the article’s normal URL, as text/markdown, at about a fifth of the size. Humans, Googlebot, and Bingbot get the identical HTML they always got — same split TIME uses, so it’s not cloaking — plus a Link: rel="alternate" header, with Vary: User-Agent so caches key correctly.
Why bother?
Because AI answers are becoming a distribution channel, and this is how you optimize for it. A model reading my page now spends its context window on my actual words instead of burning four-fifths of it on navigation and markup — which means more of the article survives into whatever ChatGPT, Claude, or Perplexity tells the person who asked. Cleaner input, better odds of being quoted correctly and cited as the source. That’s SEO for the agent era, and it costs me a fifth of the bandwidth per crawl.
What I’m not copying
The ad half. no-store plus per-request impression UUIDs exists only to make every bot fetch a billable event, and sponsor copy phrased as chatbot answers — injected where no human editor will ever see it — is a trust problem wearing a disclosure label. TIME is probably right that when most of your readers are models, someone will sell that audience. I’d rather my machine readers get the same deal my human readers get: the actual content, nothing else.
The forked web is coming either way. The only decision left is whether the version you serve the machines is your content — or somebody’s ad inventory.
