Evaluation

Pre-release: measures deployed model behavior with deterministic grading instead of treating a model name as a benchmark.

Concluded experiment: follow-up tests showed the apparent tool-call problem was mostly a serving-stack artifact.

incident-response-agent started as a hand-rolled ReAct agent. Typed state machine, separate budgets for iterations and tool calls, cycle detection, sanitized traces. That was v0. It taught me the problem space. It was also the last agent harness I’m going to build myself. For this project, the harness had become commodity infrastructure rather than the differentiating work. This is what happened after I stopped. Where v0 led The v0 loop wrapped observe → decide → act → reflect with separate execution budgets and a cycle detector. It had a trace recorder from the start: trace_id, transition events, sanitized events with no raw prompts or secrets. It worked, and it ate real sessions: malformed tool calls, budget exhaustion, repeated actions, knowing when to stop. All solved problems, all still a slog to build. ...

I set out to answer a simple question: does wrapping a model in an orchestration loop improve its answers? The idea was to put the scientific method around the model: generate an answer, test it, inspect the failure, and revise. I never got a clean answer. I learned something more useful instead: I couldn’t govern even this small system until I could independently verify its behavior. Several apparent model failures were something else: a token ceiling, a serving default, an uneven comparison, or a bug in my control flow. The score told me that a model had failed. It rarely told me which part of the system had failed or why. ...