Episode X of Phase One: Agent traffic, llms.txt, and the solitude problem
Tenth episode of the AI & Docs podcast series is up! In this one, Tom and I talk with Ethan Palm, the only technical writer at Mintlify, about what changes when nearly half the traffic to a docs site stops being human: hidden directories and content negotiation for agents, why telling an LLM “if you are an agent, read this” does nothing at all, whether llms.txt will ever become a real standard, documentation forensics, analytics that JavaScript can no longer see, and the quiet social cost of reaching for a terminal instead of a colleague.
You can watch / listen to the episode here:
Some of the things I said:
On telling agents that you’re talking to them
There’s this trend in README files and docs out there where the docs specifically address the LLMs. They say something like, “If you are an agent, read this.” I’ve done some tests and it doesn’t seem to make any difference from just stating the content itself. If you tag it as important, it’s just the same effectiveness. The LLM doesn’t really care if you’re calling them by their name.
On llms.txt and AI friendliness signaling
There are some people that say llms.txt is like — well, they declared MCP dead, and it isn’t dead. Now llms.txt is experiencing the same thing. They say it’s AI friendliness signaling. I think the cost-opportunity is still very good. It’s nice to have an llms.txt file. We still don’t know if it’s going to be read, but I think it would be nice to see them adopting a standard there.
On analytics that JavaScript can’t see
All docs sites I know have been impacted in some way or another in their analytics data, either because of AI previews in search engines or because of built-in knowledge of the models that scrape the docs. Agents accessing sites aren’t going to get tracked by traditional analytics that rely on JavaScript. So should we rather rely on server hits, for example, or other measurements?
On why ads might be what finally gives us measurement
If ads are going to become a thing in frontier AI models, with ads you will also need tools. Like it happened with search engines back in the days with SEO: you want to put an ad, you want to know the performance of that ad, and from that data indirectly you can then get to the SEO indicators.
On whether models will learn to prefer human writing
The cutoff date of most LLMs right now is quite far in the past. And this crazy estimate that about a third of all the content out there is AI-generated in some capacity — that’s a lot. It would be interesting in the future to see if you have the same doc site, A and B, one with AI-generated content not edited and the other edited, and see which one the LLM tends to prefer. I think it would be intelligent for frontier models to prefer the human-written content and the high-signal content, but we’ll see what happens there.
On solitude and why we reach for AI
I think this hits the hardest for all of us who work remotely. In an office you would validate an idea or explore a topic during a coffee chat or water cooler time. When you are on your own and maybe you work on a different time zone and you don’t have your colleagues, the decision makers, nearby — or you don’t want to interrupt them because remote work is full of focus time and it feels kind of awkward to try to create a social occasion just on Zoom. I think we tend to reach out more to AI in those occasions. There’s a direct relation, I think, between our solitude and how much we do that. And that’s perhaps the thing that unsettles me the most about using AI.
On AI that tells you to go touch grass
Project Nope is a pun, of course, but it’s an observatory for human-AI relationships. They have a PhD in psychology on the team, and essentially what they’re doing is analyzing how conversations go from a human well-being perspective. It’s understanding, for example, is this going awkward, or is the human feeling stress? I once predicted that when you work with LLMs, maybe in a few years’ time, they will tell you to stop and maybe go touch some grass, if they detect signs of stress. So I’m hoping we’ll get there somehow.
On vintage models
Some of the latest models are not great at writing, especially creative writing. And now we’re in this situation where some people, for example at work, are recommending older models. It’s like going back to the vintage and saying, “Well, that model used to write well.” So here we are trying to understand and maybe even rank the quality of writing in models.
On showing the pedigree of docs
At some point we’ll probably have to show the pedigree of AI-edited or AI-powered content too. Some factories do that: they walk you through the factory, they show you all the quality control, all the high-quality ingredients they feed to the machine. So maybe they will have to have a link in the footer saying, “This is how we produce our docs,” and you know the whole story.