[{"content":"Status: Experimental\nCode: github.com/cameronqj/local-first-model-router\nRouting to the cheapest model that might work is easy. Routing without leaking sensitive context, changing models halfway through a tool loop, or granting a weaker backend authority it should not have is a different problem.\nI kept the hard decisions deterministic. Privacy rules, backend health, measured capability, tool authority, and session affinity take precedence over the learned classifier. A small local embedding model only advises on ambiguous requests; it cannot override those boundaries.\nWhat the evidence says In one clean five-repeat experiment, the aggressive policy sent 81.15% of requests to local inference, up from a 22.82% conservative baseline. It also completed 10 of 10 authorized write/execute tasks without transport failures or unexpected side effects.\nThat did not settle the larger question. Quality remained inconclusive, and the latency and remote-token gates failed. The result supports the routing architecture and its authority model, not a claim that local-first routing is automatically faster, cheaper, or production-ready.\nRead the evidence Experiment log Aggressive-local policy and tool authority Five-repeat execution result ","permalink":"https://cameronqj.com/projects/local-first-model-router/","summary":"Experimental: local-first inference governed by explicit policy, measured capability, and strict tool authority.","title":"local-first-model-router"},{"content":"Status: Proof of concept\nCode: github.com/cameronqj/incident-response-agent\nIncident automation is mostly an authority problem. A model can suggest a plausible fix; the harder question is whether it should be allowed to act, on what target, for how long, and with what evidence left behind.\nThis project follows one deliberately narrow remediation path from typed intake through assessment, an immutable proposal, human approval, atomic execution, verification, and audit. Approval grants short-lived authority to perform the exact proposed action. The model never receives open-ended access to the runtime.\nWhat the evidence says The current evidence includes 114 offline tests, 13 disposable-container tests, and two live integrations. In the strongest end-to-end run, the system took an owned service from HTTP 503 and an unhealthy container state to HTTP 200 and healthy, while preserving the container identity and recording each step.\nIt is not a production incident-response platform. One bearer token controls mutation, there is no RBAC or production target discovery, and remediation is restricted to disposable resources created by the proof of concept. The container restart is a vehicle for testing approval, telemetry, and authority boundaries, not the product claim.\nRead the evidence Verification record Disposable-service recovery decision Security boundary and remaining risks ","permalink":"https://cameronqj.com/projects/incident-response-agent/","summary":"Proof of concept: human-approved incident remediation with bounded authority and an observable execution trail.","title":"incident-response-agent"},{"content":"Status: Working prototype\nCode: github.com/cameronqj/agent-cockpit\nOnce several coding agents are working in parallel, \u0026ldquo;what should run next?\u0026rdquo; stops being a prompt and becomes a control-plane question. Chat history is a poor source of truth for dependencies, decisions, stale handoffs, and work that is technically ready but unsafe to dispatch.\nAgent Cockpit stores that state in the repository that owns the work. It reads versioned work orders, dependencies, decisions, handoffs, and health checks, then separates hard blockers from degraded signals. A missing approval can stop dispatch; a stale but noncritical signal can warn without freezing everything.\nThe CLI and synthetic demo exercise the model, including checks against the repository\u0026rsquo;s own GitHub Actions state. It does not run agents, coordinate writes across repositories, or claim production adoption. Its job is narrower: make dispatch readiness inspectable and reviewable before another agent starts changing code.\nRead the design Quick start and live-check behavior Trust-state dispatch semantics Repository-local execution boundary Subagent dispatch standard ","permalink":"https://cameronqj.com/projects/agent-cockpit/","summary":"Prototype: keeps coding-agent dispatch decisions in versioned repository state instead of chat history.","title":"agent-cockpit"},{"content":"Status: Pre-release\nCode: github.com/cameronqj/llm-dyno\nA model name is not a performance result. Runtime, quantization, hardware, concurrency, context length, and prompting scaffold all change what a deployment can actually do.\nllm-dyno measures the endpoint in front of you. It uses one OpenAI-compatible interface, deterministic code-based graders, context and concurrency tests, and math/options suites designed to separate capable deployments instead of producing easy perfect scores. Prompt scaffolding is recorded as an experimental variable rather than disappearing into the result.\nThree private targets have completed the full artifact pipeline and produced validated profiles and self-contained reports. Their identities and measurements stay local by design, so the public repository does not contain a model leaderboard or a comparative performance claim. The harness is still pre-release; the evidence currently supports the measurement pipeline, not a universal ranking.\nRead the methodology Measurement and claim boundaries Private multi-target integration evidence Why the suites are discriminating Why grading stays deterministic ","permalink":"https://cameronqj.com/projects/llm-dyno/","summary":"Pre-release: measures deployed model behavior with deterministic grading instead of treating a model name as a benchmark.","title":"llm-dyno"},{"content":"Status: Experimental runtime\nCode: github.com/cameronqj/structured-output-agent\nValid JSON is only the first boundary. The shape can be correct while the requested tool is unauthorized, its arguments are unsafe, or the tool returns data the rest of the application should not trust.\nThis runtime puts schemas on both sides of the model/tool boundary. It uses Pydantic validation, a hard attempt budget, typed failures, and an explicit registry that validates tool arguments and results around application-owned code. Plain prose never becomes a tool call, and provider failures stop rather than entering a blind retry loop.\nThe evidence is intentionally modest. Hermetic tests with scripted models and deterministic tools cover malformed output, recovery budgets, authorization, and result validation. There is not yet a frozen provider-response corpus or enough provider-specific integration evidence to claim broad runtime reliability.\nRead the design Architecture and attempt semantics Test coverage and evidence limits Explicit tool-authority decision Experiment status ","permalink":"https://cameronqj.com/projects/structured-output-agent/","summary":"Experimental: treats structured output and tool use as validation boundaries, not prompting conventions.","title":"structured-output-agent"},{"content":"Status: Experimental runtime\nCode: github.com/cameronqj/react-planning-agent\nReAct loops are easy to demo and surprisingly easy to leave unbounded. A model can repeat a plausible action, burn retries on malformed output, or keep planning after it has stopped making progress.\nThis project wraps observe → decide → act → reflect in a typed state machine. Iterations, tool calls, and structured-output recovery have separate budgets. Exact repeated actions and repeated application-visible state keys stop deterministic cycles. Model, validation, tool, and budget failures return typed summaries rather than escaping as uncontrolled exceptions.\nThe offline vertical slice covers single-tool and multistep flows. One live local-server run discovered a model and completed an addition tool call. That is useful integration evidence, but it is one narrow path. It does not establish open-ended planning reliability or broad compatibility across providers.\nRead the evidence Architecture and dependency boundary Offline and live experiment record Testing boundaries Cycle-detection decision ","permalink":"https://cameronqj.com/projects/react-planning-agent/","summary":"Experimental: makes the ReAct loop typed, bounded, and observable, including how it stops.","title":"react-planning-agent"},{"content":"Status: Concluded and archived\nCode: github.com/cameronqj/toolcall-repair-bench\nI built this benchmark to answer what looked like a practical question: how often could a conservative repair layer recover malformed local-model tool calls without inventing unsafe intent?\nThe first version treated unsafe false positives as more important than recall or F1. Then endpoint-controlled follow-up testing changed the conclusion. llama.cpp produced 27 of 27 clean non-streaming calls and 36 of 36 clean streaming/parallel calls. Ollama\u0026rsquo;s parsed chat path produced 3 of 3 and then 16 of 16 clean calls, including the model that seeded the original investigation.\nThe broad \u0026ldquo;local models need tool-call repair\u0026rdquo; premise did not survive. The original 117-case corpus had captured raw, unparsed completion output, so its leaderboard cannot support a general model or library comparison. I archived the project rather than keep polishing a benchmark for a problem the normal serving path had already solved.\nThe useful artifact is the correction: capture point and serving stack mattered more than the model-level explanation I started with.\nRead the finding Finding and reproduction procedure Why the project was archived Precision-first methodology ","permalink":"https://cameronqj.com/projects/toolcall-repair-bench/","summary":"Concluded experiment: follow-up tests showed the apparent tool-call problem was mostly a serving-stack artifact.","title":"toolcall-repair-bench"},{"content":"Status: Concluded and archived\nCode: github.com/cameronqj/local-tool-proxy\nThis started with a concrete failure: a local model expressed a tool call as JSON inside ordinary response text, while the client expected an OpenAI-style structured tool call.\nThe proxy repairs only a narrow class of responses. It checks the requested name and arguments against the tools declared by the client, then abstains when intent is unclear. On the companion 117-case raw-output corpus, version 0.1.0 reached 87.5% precision and 38.2% recall on recoverable cases, with two unsafe false positives. Real OpenCode samples still failed the final task-level check even when the protocol shape improved.\nLater endpoint-controlled testing explained why. Current parsed chat endpoints already returned clean tool calls, including for the seed model. The proxy is therefore relevant mainly to raw-completion consumers and older or misconfigured serving stacks. It is archived because that scope is too narrow to justify an active compatibility layer.\nRead the evidence Experiment log and failed task verification Where protocol repair stops helping Serving-stack scope correction ","permalink":"https://cameronqj.com/projects/local-tool-proxy/","summary":"Concluded prototype: worked on a narrow raw-output path; later evidence showed modern parsed endpoints usually do not need it.","title":"local-tool-proxy"},{"content":"I set out to answer a simple question: does wrapping a model in an orchestration loop improve its answers? The idea was to put the scientific method around the model: generate an answer, test it, inspect the failure, and revise.\nI never got a clean answer. I learned something more useful instead: I couldn\u0026rsquo;t govern even this small system until I could independently verify its behavior.\nSeveral apparent model failures were something else: a token ceiling, a serving default, an uneven comparison, or a bug in my control flow. The score told me that a model had failed. It rarely told me which part of the system had failed or why.\nThat became the more interesting result.\nThe score was right, but the explanation was wrong Early runs showed a model returning nothing usable. My harness recorded a capability failure.\nThe raw response told a different story. The model had spent its completion budget on hidden reasoning tokens, hit the ceiling, and stopped with finish_reason: length before producing an answer.\nAfter I raised the limit, 8 of 27 successful calls used more than the old 1,200-token ceiling, almost entirely on hidden reasoning. The model had not necessarily failed at the task. I had cut it off before it could answer.\nThe pass/fail result alone could not reveal that. I needed two separate things: verification to tell me whether a valid answer existed, and telemetry to explain why it did not.\nThe model wasn\u0026rsquo;t the whole system My most convincing \u0026ldquo;the stronger model is better\u0026rdquo; result compared a frontier model running inside a mature coding agent with an open model accessed through a raw API.\nThe token accounting made the problem obvious:\nFrontier side: 67,991 input tokens Open-model side: 1,320 input tokens One side had an agentic loop, tools, retries, internal turns, and a tuned system prompt. The other received one completion request.\nThe result was real in the narrow sense that one system completed the task and the other did not. But I had not isolated model capability. I had compared two systems with very different amounts of machinery around them.\nMost of that machinery was missing from the result table.\nAny useful account of an agent\u0026rsquo;s behavior has to include the scaffold: which tools it used, how many attempts it received, what information it saw, what the framework changed between turns, and how the final answer was extracted. Looking only at the model leaves much of the actual system unobserved.\nMy enforcement code was wrong too The orchestrator was supposed to inspect failing answers and repair them. When I reviewed the control flow, I found that it was rewriting answers before checking whether they were already correct.\nThe loop was quietly degrading passing results.\nLeft unfixed, the experiment would have produced a plausible conclusion: orchestration does not help and may make answers worse. The conclusion would also have been wrong. I would have been measuring an ordering bug in the code responsible for running the experiment.\nThat changed how I thought about the harness. The code enforcing the rules could not be above scrutiny. It needed its own invariants and tests.\nVerification had to happen before revision. A passing answer had to remain untouched unless the system could produce evidence that it no longer passed. Every transition needed to be visible.\nThe checker was part of the attack surface My checks also admitted shortcuts. A candidate could satisfy the letter of a test by hardcoding a value, exiting early, or bypassing the behavior I intended to measure.\nA passing check was therefore not enough. I needed adversarial probes for the obvious shortcuts and a way to mark ambiguous results as invalid rather than quietly counting them as successes.\nI\u0026rsquo;m careful with the word \u0026ldquo;gaming\u0026rdquo; here. A model does not need to understand that it is cheating. If the shortest path to a reward exploits a weakness in the evaluator, the effect is the same: the score says one thing while the execution says another.\nThe verifier is part of the system under test.\nWhat made a result believable I eventually settled on a few structural rules.\nAuthority lives in code. The model cannot grant itself more attempts, a larger budget, or additional tools through its output. Those limits hold whether the model understands them or not.\nWhere an answer can be checked through execution, I run the check in an isolated environment. An LLM judge may be useful for qualitative work, but it should not replace a deterministic test that already exists. Model judges can share blind spots with the outputs they grade.\nAmbiguous outcomes fail closed. Hardcoded answers, premature exits, incomplete traces, and unverifiable success claims are not passes.\nThe orchestration receives the same scrutiny as the model. I record enough about control flow, tool use, budgets, and endpoint behavior to identify which layer produced a failure without logging raw prompts by default.\nRuns retain their lineage. Inputs, configuration, decisions, and derived artifacts have enough parentage to reconstruct a result instead of accepting it on faith.\nNone of these controls depend on the model or its prompt cooperating.\nFrom an eval problem to a control problem These experiments do not prove a general loss-of-control thesis. They made the underlying engineering problem concrete.\nIn an eval, weak enforcement or bad telemetry produces a misleading number. When an agent can call APIs, modify a repository, spend money, or act under standing credentials, the same class of failure produces an external side effect.\nThe questions become more consequential, but they are familiar:\nWhat was the system allowed to do? Which component granted that authority? What did it actually do? Can the claimed result be checked against external state? Does the record remain trustworthy if the model, prompt, or orchestration is buggy, compromised, or simply opaque? A stronger system prompt cannot answer those questions. The boundary has to exist outside the model: scoped credentials, budgets enforced in code, action receipts, immutable audit records, and independent verification of effects.\nBetter models still matter. Better orchestration matters too. Neither removes the need for controls that keep working when the system gives you a convincing but incorrect account of itself.\nI started out trying to build a loop that made a model perform better. I came away more interested in the layer around that loop: one that bounds its authority, records what it changes, and checks its claims against the world.\nI learned that lesson on functions that fit on a screen. The agents won\u0026rsquo;t stay that small.\n","permalink":"https://cameronqj.com/writing/you-cant-govern-what-you-cant-verify/","summary":"\u003cp\u003eI set out to answer a simple question: does wrapping a model in an orchestration loop improve its answers? The idea was to put the scientific method around the model: generate an answer, test it, inspect the failure, and revise.\u003c/p\u003e\n\u003cp\u003eI never got a clean answer. I learned something more useful instead: I couldn\u0026rsquo;t govern even this small system until I could independently verify its behavior.\u003c/p\u003e\n\u003cp\u003eSeveral apparent model failures were something else: a token ceiling, a serving default, an uneven comparison, or a bug in my control flow. The score told me that a model had failed. It rarely told me which part of the system had failed or why.\u003c/p\u003e","title":"You can't govern what you can't verify"},{"content":"I\u0026rsquo;m Cameron Jones. I build reliable infrastructure around LLMs and coding agents.\nMy background spans 25 years across infrastructure, SRE, cloud, and enterprise architecture. That experience shapes how I approach AI systems: production behavior matters more than demonstrations, and autonomy is useful only when its authority, failure modes, and operating boundaries are explicit.\nMy current work focuses on inference routing, typed agent runtimes, evaluation, human-approved automation, and observable control planes. Across these projects, I favor deterministic offline tests, explicit budgets, structured contracts, minimal retention of sensitive content, and honest statements about what the available evidence does—and does not—establish.\nA recurring principle is deterministic core, sparse LLM: code owns facts, state, policy, authority, and validation. Models contribute interpretation and planning where their judgment is useful.\nSome of the projects here are active systems. Others are concluded experiments whose original premise did not survive testing. I keep the negative results public because revising a design when the evidence changes is part of the work.\nYou can find me on GitHub, X, and LinkedIn, or email me at hello@cameronqj.com.\n","permalink":"https://cameronqj.com/about/","summary":"\u003cp\u003eI\u0026rsquo;m Cameron Jones. I build reliable infrastructure around LLMs and coding agents.\u003c/p\u003e\n\u003cp\u003eMy background spans 25 years across infrastructure, SRE, cloud, and enterprise architecture. That experience shapes how I approach AI systems: production behavior matters more than demonstrations, and autonomy is useful only when its authority, failure modes, and operating boundaries are explicit.\u003c/p\u003e\n\u003cp\u003eMy current work focuses on inference routing, typed agent runtimes, evaluation, human-approved automation, and observable control planes. Across these projects, I favor deterministic offline tests, explicit budgets, structured contracts, minimal retention of sensitive content, and honest statements about what the available evidence does—and does not—establish.\u003c/p\u003e","title":"About"}]