I never got a clean answer. I learned something more useful instead: I couldn’t govern even this small system until I could independently verify its behavior.

Several apparent model failures were something else: a token ceiling, a serving default, an uneven comparison, or a bug in my control flow. The score told me that a model had failed. It rarely told me which part of the system had failed or why.

That became the more interesting result.

The score was right, but the explanation was wrong

Early runs showed a model returning nothing usable. My harness recorded a capability failure.

The raw response told a different story. The model had spent its completion budget on hidden reasoning tokens, hit the ceiling, and stopped with finish_reason: length before producing an answer.

After I raised the limit, 8 of 27 successful calls used more than the old 1,200-token ceiling, almost entirely on hidden reasoning. The model had not necessarily failed at the task. I had cut it off before it could answer.

The pass/fail result alone could not reveal that. I needed two separate things: verification to tell me whether a valid answer existed, and telemetry to explain why it did not.

The model wasn’t the whole system

My most convincing “the stronger model is better” result compared a frontier model running inside a mature coding agent with an open model accessed through a raw API.

The token accounting made the problem obvious:

• Frontier side: 67,991 input tokens
• Open-model side: 1,320 input tokens

One side had an agentic loop, tools, retries, internal turns, and a tuned system prompt. The other received one completion request.

The result was real in the narrow sense that one system completed the task and the other did not. But I had not isolated model capability. I had compared two systems with very different amounts of machinery around them.

Most of that machinery was missing from the result table.

Any useful account of an agent’s behavior has to include the scaffold: which tools it used, how many attempts it received, what information it saw, what the framework changed between turns, and how the final answer was extracted. Looking only at the model leaves much of the actual system unobserved.

My enforcement code was wrong too

The orchestrator was supposed to inspect failing answers and repair them. When I reviewed the control flow, I found that it was rewriting answers before checking whether they were already correct.

The loop was quietly degrading passing results.

Left unfixed, the experiment would have produced a plausible conclusion: orchestration does not help and may make answers worse. The conclusion would also have been wrong. I would have been measuring an ordering bug in the code responsible for running the experiment.

That changed how I thought about the harness. The code enforcing the rules could not be above scrutiny. It needed its own invariants and tests.

Verification had to happen before revision. A passing answer had to remain untouched unless the system could produce evidence that it no longer passed. Every transition needed to be visible.

The checker was part of the attack surface

My checks also admitted shortcuts. A candidate could satisfy the letter of a test by hardcoding a value, exiting early, or bypassing the behavior I intended to measure.

A passing check was therefore not enough. I needed adversarial probes for the obvious shortcuts and a way to mark ambiguous results as invalid rather than quietly counting them as successes.

I’m careful with the word “gaming” here. A model does not need to understand that it is cheating. If the shortest path to a reward exploits a weakness in the evaluator, the effect is the same: the score says one thing while the execution says another.

The verifier is part of the system under test.

What made a result believable

I eventually settled on a few structural rules.

Authority lives in code. The model cannot grant itself more attempts, a larger budget, or additional tools through its output. Those limits hold whether the model understands them or not.

Where an answer can be checked through execution, I run the check in an isolated environment. An LLM judge may be useful for qualitative work, but it should not replace a deterministic test that already exists. Model judges can share blind spots with the outputs they grade.

Ambiguous outcomes fail closed. Hardcoded answers, premature exits, incomplete traces, and unverifiable success claims are not passes.

The orchestration receives the same scrutiny as the model. I record enough about control flow, tool use, budgets, and endpoint behavior to identify which layer produced a failure without logging raw prompts by default.

Runs retain their lineage. Inputs, configuration, decisions, and derived artifacts have enough parentage to reconstruct a result instead of accepting it on faith.

None of these controls depend on the model or its prompt cooperating.

From an eval problem to a control problem

These experiments do not prove a general loss-of-control thesis. They made the underlying engineering problem concrete.

In an eval, weak enforcement or bad telemetry produces a misleading number. When an agent can call APIs, modify a repository, spend money, or act under standing credentials, the same class of failure produces an external side effect.

The questions become more consequential, but they are familiar:

• What was the system allowed to do?
• Which component granted that authority?
• What did it actually do?
• Can the claimed result be checked against external state?
• Does the record remain trustworthy if the model, prompt, or orchestration is buggy, compromised, or simply opaque?

A stronger system prompt cannot answer those questions. The boundary has to exist outside the model: scoped credentials, budgets enforced in code, action receipts, immutable audit records, and independent verification of effects.

Better models still matter. Better orchestration matters too. Neither removes the need for controls that keep working when the system gives you a convincing but incorrect account of itself.

I started out trying to build a loop that made a model perform better. I came away more interested in the layer around that loop: one that bounds its authority, records what it changes, and checks its claims against the world.

I learned that lesson on functions that fit on a screen. The agents won’t stay that small.