Loop Engineering, Chapter 8: Comprehension Debt and Model Collapse

This is the final post in a series walking through my book Loop Engineering: Scaling and Governing Agentic AI. In the previous chapter, we scaled judgment with automated evaluation. This one closes the book with the two risks that only come due at scale.

Everything in the book pushes toward loops that run faster, more autonomously, and at greater scale than a human could manage by hand. That is the point. But every capability carries a cost that does not show up in any single run and only becomes visible after the system has been running well, at speed, for a while. The final chapter is about two such costs, and they are dangerous precisely because the system that incurs them looks healthy right up until it does not.

The first is comprehension debt. When an agent can write, test, and merge code faster than any human can read it, the codebase fills with working software that nobody actually understands. It is the cousin of technical debt, but worse in one specific way: with technical debt, someone understood the trade-off when they made it. With comprehension debt, the code is clean and the tests are green and no human ever built a mental model of it. The bill comes due at two in the morning, when an engineer is asked to debug a stranger's work and the stranger was an agent that is no longer in the room.

When models eat their own output

The second risk operates on a longer timescale. Model collapse — also called Model Autophagy Disorder, or AI cannibalism — is what happens when models train on data generated by other models, recursively, across generations. Each generation learns from a slightly narrower, slightly more biased approximation of reality than the last, like a photocopy of a photocopy. The symptoms are loss of variance, erosion of the rare cases in the tails, and amplification of whatever bias was already present. The early generations look fine, which is exactly why it gets ignored until it is expensive.

Neither risk is a bug in any single component, and neither can be caught in a unit test. The chapter's answer is to treat human comprehension as a resource you allocate deliberately — spending it on the parts of the system that matter and accepting debt knowingly elsewhere — and to keep models anchored to fresh, human-originated data, with strict provenance and adversarial checks that watch for the narrowing before it compounds. The book formalizes the drift bound that says exactly how much fresh data you need to keep collapse in check.

That closes the series and the book. We went from the anatomy of a single loop to the primitives that compose it, the frameworks that orchestrate it, the operations that run it, the governance and human controls that keep it accountable, the evaluation that lets it judge itself, and the slow risks that come for it at scale. The rest is the part no book can hand you: the judgment to apply it to a system you actually own. Thank you for reading along.