Status: Concluded and archived
Code: github.com/cameronqj/toolcall-repair-bench
I built this benchmark to answer what looked like a practical question: how often could a conservative repair layer recover malformed local-model tool calls without inventing unsafe intent?
The first version treated unsafe false positives as more important than recall or F1. Then endpoint-controlled follow-up testing changed the conclusion. llama.cpp produced 27 of 27 clean non-streaming calls and 36 of 36 clean streaming/parallel calls. Ollama’s parsed chat path produced 3 of 3 and then 16 of 16 clean calls, including the model that seeded the original investigation.
The broad “local models need tool-call repair” premise did not survive. The original 117-case corpus had captured raw, unparsed completion output, so its leaderboard cannot support a general model or library comparison. I archived the project rather than keep polishing a benchmark for a problem the normal serving path had already solved.
The useful artifact is the correction: capture point and serving stack mattered more than the model-level explanation I started with.