Loop Engineering, Chapter 7: Automated Evaluation and LLM-as-a-Judge

This is Part 7 of a series walking through my book Loop Engineering: Scaling and Governing Agentic AI. In the previous chapter, we designed the human into the loop. This one is about scaling judgment beyond what any human loop can sustain.

The case for automated evaluation is, at bottom, an argument about cost per judgment. A human reading an agent's output, forming an opinion, and recording a score is slow and expensive, and that expense scales linearly with volume. That is fine when you ship once a quarter and a panel of experts can sit for an afternoon. It collapses the moment you want to evaluate continuously, on every commit, across thousands of cases, several times a day. A capable model used as a judge reaches roughly 80% agreement with human raters at something like 500 times lower cost — and that ratio is what turns evaluation from a periodic event into a continuous gate.

The chapter sorts judges into three shapes and is clear about when each applies. A single-output referenceless judge scores an answer against a rubric with no gold standard to compare to. A single-output reference-based judge scores it against a known-good answer. A pairwise judge does the easiest thing of all — it picks the better of two candidates — and is often the most reliable because relative judgments are easier than absolute ones.

The judge has biases too

Handing evaluation to a model imports a new set of failure modes, and pretending otherwise is how teams ship a judge that quietly lies to them. Position bias makes a judge favor whichever candidate it sees first; you correct it by swapping the order and looking for agreement. Verbosity bias makes it reward length as a proxy for quality; you control for it explicitly. Self-preference bias makes a model favor text in its own style; you correct it with cross-model evaluation, so a model is not grading its own kind of homework.

That last idea is G-Eval: instead of taking a single integer, you weight each possible score by the probability the model assigns it, producing a continuous score with far more resolution. The chapter pairs it with directed acyclic graphs of deterministic checks — for the parts of quality that should not be left to a judge's discretion at all — and shows a judge prompt and a token-probability scoring snippet so the technique is concrete. The underlying formula is worked out in the book's appendix.

Tomorrow: the final chapter, and the two risks that only show up after the system has been running well, at speed, for a while — comprehension debt and model collapse.