Frozen write-up from 2026-08 — numbers as of that date; the README scoreboard is current.
agent-sandbox
This repo turns that into a number. It also caught its own scoring system lying, and fired it.
Five small office agents on local models (4–14B, one 16 GB laptop). The agents are deliberately boring — a prompt, a tool list, one config line. The asset is the exam suite around them: 86 committed cases, deterministic scoring, no LLM judge anywhere. This page is the visual companion to the repo's frozen write-up; the failures stay on it.
scroll · the exam, the gate, the sweep, and what keeps them honest ↓
what you're looking at
The agents are replaceable in an afternoon. What compounds is the discipline around them. This page shows the system first, and the method that discipline enforces runs through every beat.
runner · gate · router · watcher
the habits the numbers come from
Here's how it works.
↓per-case rows: the committed 2026-07-24 snapshot · the suite has since grown to 86 cases across 5 exams · ⚠ a perfect score is a broken exam, not a good model: the sweep found email sorting's unseen saturated — every model swept scored 4/4
agents are replaceable · the exam suite is the asset that outlives them
current: 5 agents · 86 committed cases · 20 test files · no LLM judge anywhere in the suite · why steps matter: ten steps at 90% each only works 35% of the time
01 · the exam
An agent here is small: a prompt, its tools, one model line in a config file. What makes it real is the exam it is born with. A fixed set of cases, scored by code, with the failures left on the page. Some cases the agent practices on. Some it has never seen, and those are the ones that count.
The grading gets stricter as the task gets harder: exact labels, then per-field partial credit, then code-checked properties where the wording can vary, up to full multi-step tool runs, where it matters which tools were called, how many steps it took and whether the answer is grounded. Chained steps compound: ten steps at 90% each is a 35% workflow.
◆ 86 committed cases across 5 agents, every check plain code. Both real failures stay on this page.
the passing figures are real saved results · the drift is staged on purpose
— now poison one prompt, on purpose —
blocked · a slipped score stops the release, it does not warn about it
the customer follow-up figures are a real saved result: practice 3 of 3 · unseen 3 of 4 · the other rows come from the 2026-07-23 comparison run · only the failing row is simulated, and the mechanics around it are real
02 · the gateruns on every commit · exit 1
Every score an agent passes with is saved. Before anything ships, every saved score is measured again. If one of them has slipped, the release stops. Red means stop, and no AI's opinion can override it. The questions the agent has never seen stay that way: you may tune an agent on the ones it practices with, never on the ones it is graded by.
Nothing passes silently either. An empty answer raises. An answer the checks cannot read fails. A broken check fails. Loud, never a placeholder.
◆ A change that lowers a score stops the release on its own. Nobody has to be watching.
unseen only · reply drafting column withheld (the grader was too unreliable to trust) · llama3:8b = fastest usable · qwen3:4b = long pole · fast and wrong is still wrong
how to read the differences · the current model ran every question five times first, and that spread is the normal variation
current model 0.6 over five runs · this question answers differently run to run
difference inside normal variation — dismissedcurrent model fails all five runs, every time · needs a chained second tool call · upgrade candidate, decided by PR, not by this page
gemma4:e4b-it-qat passes — outside the band · a real gaprunning it five times catches a model disagreeing with itself · it will never catch one being confidently wrong, which is why the checks are plain code
comparison run, 2026-07-23, on one laptop · the look-up question passes 3 times in 5, which is the normal variation · the two-step question fails every time · the reply drafting column is left out because its grader was too unreliable to trust, see beat 04
03 · the sweep
Seven models, four tests, 72 minutes start to finish on one laptop. About a lunch hour, no cluster, no cloud bill. Before any of it, the current model runs every question five times. Some questions come back with a different answer each time, and that spread is the normal variation: a challenger that only improves by less than that has not improved, however good the screenshot looks.
What survives is signal. The current model fails the two-step question every time, five runs out of five. gemma4:e4b-it-qat passes it, the only model in the comparison that does, and it matched or beat the current model everywhere else. That gap is bigger than the normal variation, so it is a real capability rather than a coin flip. Swap or keep becomes a measurement instead of a debate.
◆ Seven models compared in 72 minutes on one laptop. One of them passes a question the current model has never passed.
04 · the audit
We used to let an AI grade some answers against a written scoring guide. Rewording that guide, with the same model, the same questions and the same answers, moved the score from one end of the range to the other. The number was measuring our wording, not the agent. A grader you can't trust corrupts every measurement, so the AI grader went: every check in the suite is now plain code.
The same comparison run showed that two of the four tests had quietly stopped telling models apart. One had become too easy, because every model aced it. One was drowned by the grader's own noise. Finding that out is what a measurement layer is for.
◆ Rewording the grading instructions moved the score from 2 of 4 to 4 of 4 on identical answers. That is why no AI grades anything here.
also caught: every model scored 4 of 4 on email sorting · a test has to be re-checked for whether it is still hard enough, not just for whether it is right
the same answers, graded twice: the old wording scored 2 of 4 twice, the new wording 4 of 4 three times, on identical replies · the judge also mis-passed 3 of 3 invented-date replies before it went · no LLM judge anywhere in the suite, by decision, 2026-07-24 · the cost is written down: a generic reply now passes reply drafting — accepted, documented, on the fix list
That was the measuring layer.
How swapping the model becomes one setting, a corrections loop that is honestly one trace deep, two experiments that died cheap on purpose, the next instrument being built — and what is broken right now, stated plainly.
↓→ re-running all 86 cases against the new one · same bar
a new backend is one line in a settings table, never new code · swap or keep, the numbers decide
EU tier verified 2026-07-18: Scaleway (Paris-only DC) · OVHcloud AI Endpoints · Nebius · the swap shown illustrates real mechanics, and no current model was moved on this page
05 · the router
Swapping the model under an agent is one line of configuration, never new code. Three tiers, chosen per agent by how sensitive its data is: models on your own machine for anything touching personal data, EU-hosted open models when a task outgrows the laptop, and the big commercial APIs as an opt-in escape hatch for the genuinely hard.
The measuring travels with the swap. Change the line, rerun all 86 cases, read the difference. Sovereignty without measurement is just a different place to guess.
◆ Changing which model runs an agent is one line. All 86 cases rerun against it before the change is kept.
06 · the loopdesigned · not yet fed
Every live run is recorded. When one goes wrong, the wrong answer plus a hand fix becomes a new committed case, permanently, so the same mistake cannot quietly come back. The exam gets harder exactly where reality disagreed with it.
Honestly: this is the least-built part of the system. The shadow logger shipped on 2026-07-24 and the record still holds exactly one trace a month later — the loop is designed and wired, not yet fed. It stays on this page because the page's rule is that unflattering numbers stay in.
◆ A wrong answer plus a hand fix becomes a permanent case. The record: one trace, unchanged since July — stated, not hidden.
the record has not moved since 2026-07-24 · designed, not yet fed · labelled as such
1 live trace, recorded 2026-07-24, still the only one as of 2026-08-24 (results/live/, checked today) · turning a mistake into a permanent case is a manual step today, by design, because someone reviews it first
lane killed before a line of it was built · under an hour of compute
commit 0bc8c73, “step 0a RAN and KILLED the lane” · the follow-up doc records the probe itself was wrong twice in the direction its author hoped — falsifiers get audited like everything else (docs/sprint-map.md, E20)
docs/model-size-probe-2026-08-05.md · the probe nearly lied: the first run used the default 120 s timeout and returned zero completed cases — killed and rerun at 900 s rather than interpreted as capability failure
07 · experiments die cheapkill criteria · pre-registered
Before the first multi-turn exam was built, a falsifier ran against its premise — the field's claim that models drop instructions as conversations grow. Kill criterion written down in advance, nine model calls, zero code: the champion held every constraint, three turns out of three. The lane died in under an hour instead of consuming a week of exam-building on a false premise.
The same habit priced a hardware question. Would a bigger model fix the hardest exam? A 4B, an 8B and a 14B all score an identical 7 of 10 on it, failing in opposite styles. The fix is a better target, not more parameters — and nothing got bought.
◆ A pre-registered falsifier killed a planned exam for nine model calls; a 40-minute probe settled a hardware spend at €0.
08 · the next instrument
The small exams rank bad models out but cannot pull two good ones apart, so a harder instrument is being built from real code: every merged PR of a production repo, distilled through a measured funnel into tasks that reproduce red→green. When the human oracle — the author's own merged code, pushed through the identical pipeline — graded end-to-end on only 3 of 10 tasks, the count was rewritten down, and a second task shape was added until the oracle ran green 12 of 12 on the first try.
The first model through scores 0 of 9 on the strict gate — and the zero has structure: six near-misses passing 76–97% of their gating tests, three deaths where the model writes 22–41k characters and never completes a file. Named failure modes on a discriminating instrument beat the saturated 1.00s that beat 01 flags as defects.
◆ 1,150 real PRs distilled to 15 oracle-verified tasks. First model: 0/9 strict, with named failure modes — the same honesty, pointed at a harder instrument.
— then the human oracle graded the graders —
pool: 15 gradeable tasks · first model, qwen3-8b: 0 of 9 strict — 6 near-misses at 76–97% · 3 termination deaths
docs/e-corpus-sweep-2026-08-14.md · e-corpus-create-2026-08-15.md · e-corpus-ladder-2026-08-15.md · the ladder run was operator-stopped at task 10 of 14 and is titled PARTIAL, not rounded up · task #369 excluded rather than scored (prompt exceeds the context window) · the oracle passes 12/12 through the same pipeline, so the harness is not the zero
the ledger
Solid mauve is shipped and measured. Runnable from a clean clone this afternoon. Dashed amber is broken or deferred, stated plainly — the same list the repo's own write-up keeps, because knowing precisely what doesn't work yet is the method.
in the box today
5 agents, 86 committed cases, 20 test files, practice and unseen kept apart, every check plain code. No LLM judge anywhere, by decision, not by accident.
exact labels → per-field credit → code-checked properties → full multi-step tool runs, graded on the whole run rather than the final sentence.
every passing score saved · a slipped score exits 1 and stops the release · fail loud, never a placeholder · the unseen cases stay unseen.
7 models × 4 exams in 71.9 min on one laptop. Normal variation computed before any ranking, so a lucky case can't ship a model.
model swaps as one config line. EU endpoints (Scaleway · OVHcloud · Nebius) verified 2026-07-18. When exams can't separate two models it says cannot-distinguish instead of guessing.
15 gradeable tasks distilled from 1,150 merged PRs, oracle-verified 12/12, selection step committed so the funnel reruns from a clean clone.
broken or deferred · stated plainly
heldout sets are 10–11 cases, so one case ≈ 9–10% of a score — inside the noise band. The routing layer's honest headline output is a refusal: cannot-distinguish, grow heldout before revisiting.
measures hallucination + format only. When its LLM judge mis-passed 3/3 invented-date replies it was dropped, and the cost written down: a generic reply now passes.
zero cases in either, despite the target market. Deferred on purpose: adding languages to exams that can't rank buys saturated scores in more languages.
designed, wired, and one trace deep for a month. The feed that turns mistakes into cases has no traffic yet, and this page says so rather than dressing it up.