Episode VI of Phase One: Doc testing, skills files, and the guardians of knowledge
Sixth episode of the AI & Docs podcast series is up! In this one, Tom, Manny Silva, and I talk about documentation testing (the deterministic kind), skills files and why they matter, the ethical tension of externalizing your expertise, and the evolving role of tech writers as content curators. Or, if you prefer the romantic version, guardians of knowledge.
You can watch / listen to the episode here:
Some of the things I said:
On docs as a product you develop
“What Manny is describing, I think, is treating the docs like a product in itself that you are developing. And in fact, we are often developing it now because we are coding new features. We’re coding entire systems to publish the documentation or to create test harnesses.”
“And then you come up with metrics and some of those are success metrics and reliability metrics, and testing those, I think, is super important.”
On skills and why they’re documentation
“I would say — distribution is one thing. Like you want to share them. You want to version them. You want to also make them slightly more accessible to other people that might lack the context of your code base.”
“And then there’s also another aspect which I find very interesting, which is all the topping that you put onto a skill in the form of front matter. For example, whether the skill should fork the context, which saves you main context in some agent, et cetera, or what kind of tools are allowed by default. So it’s a little time saver, I would say, but they do feel a lot like the good old macros that we used to use in Office, only that this time you package them, you distribute them, other people can reuse them.”
On tech writers and the magic of prompting
“We were both thinking — like we both noticed that we are actually pretty good at prompting to create docs. I don’t know what your experience with it is, but I’ve seen the non-writers like developers, PMs trying to create documentation or curating documentation through LLMs, and they’re not pulling good work out of it. They’re not doing a good job. So there’s something in our experience or in the way we ask the machine that is valuable.”
“Sometimes we really underestimate the value we bring by asking LLMs to do things in the way we know is good.”
On guarding the knowledge
“It’s the guardians of knowledge, of the high-quality knowledge. I think that companies that will really have successful AI usage — the best skills, etc. — are the companies that will have a strongly curated, well-preserved, protected repositories of high-signal, high-quality knowledge, or context if you want. And for those you need keepers.”
“Now it’s a bit more romantic. You can picture, like the librarians of old, guarding the knowledge and taking care of the old tomes. But really, we really need machines not to touch that. We really need that high-quality signal, and extract it and use it when necessary, creating things with LLMs or even skills.”
On the MCP server and human-written docs as the final arbiter
“At work, for example, we have created this MCP server that taps into our human-written documentation. And we are using that for so many things, like for example to create new skills or validate existing skills or check for drift in things that LLMs have done. The final arbiter is that high-quality, human-made knowledge. And yeah, developers are not going to take care of that.”
On the future being both likely and survivable
“It’s sadly more likely than the disappearance of tech writing. Like, I don’t think tech writing is gonna go away, but I do think that in some cases there will be at least temporarily some reorgs happening where maybe fewer of us will be doing a similar job or orchestrating things at scale.”
“I’m hopeful though that this will be just a bump on the road. And as research catches up and finds that maybe information quality degrades over time or that you need more human intervention, and I’m sure we will reach that point at some point. But it might take some time.”