AI and IA

AI companies should hire for taste. Models already write code and are learning to improve themselves without humans in the loop. AI is quickly moving up the software development stack. Will it take on AI research next?

The critical resource of the future might not be developing AI but finding people inventive enough to design the interfaces that keep humans in the loop, people who study where understanding breaks and a person can no longer tell whether the system is right. We probably need a new word for it.

John Markoff drew this line a decade ago. In "Machines of Loving Grace" he traced two traditions since the sixties. Artificial intelligence (AI), the project of building machines that replace human work, and intelligence augmentation (IA), the project of building machines that extend it. John McCarthy on one side, Douglas Engelbart on the other. The field is fragmenting along that seam. One branch races to remove humans from tasks and could become increasingly self-accelerating. The other has the near-impossible task of keeping up.

The trouble is that the loop is a moving target. As the systems accelerate, holding a human inside it through better interfaces buys time, the way interpretability and alignment buy time, but it does not settle the question. Eventually the only way to stay in the loop is to change what a human in the loop is.

Maybe Markoff was right, and we should start treating taste, interface design, evals, alignment, mechanistic interpretability, even brain augmentation and uploading, as a single field of intelligence augmentation. We will need to use models to build the augmentation and need augmented humans to keep the models aligned, so neither is safe without the other.

But the economics cut against it. Capital pours into fast, self-improving AI, a flywheel that compounds as models build the next models removing humans from the loop. Intelligence augmentation draws orders of magnitude less, and even that grows sublinearly, so the gap does not hold at a ratio, it widens. IA needs to accelerate. Even AI models benefit from it as human bandwidth is the bottleneck.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.