Small local models, measured on office tasks

Eight agents (seven in the hosted comparison), 226 committed cases, deterministic grading with no LLM judge, every candidate sat the same held-out cases at temperature 0. How it works in four steps, then three findings, one chart, one command.

Scores are held-out cases passed, never percentages. Dollars are per 10,000 tasks on the tokens of that run at the OpenRouter rate of 2026-10-08; metered means a run of that model, modelled means arithmetic on stated inputs. The private repo holds the build ledger; everything on this page has a probe folder in the public one.

How it works, in four steps

1. Name the tasks you already send to a model. Triage, extraction, drafting, summaries. The one requirement: a person can say, case by case, whether the answer was right. A task nobody can check is not scored here, and the page says so. The eight tasks below are the demo set; a client's exam is never published.
2. Write down what a correct answer looks like. Twenty to forty cases per task with their checks: the label, the fields, the facts that must appear, the tools that must be called. A third is held out and never tuned on. No model grades another model's answer, anywhere. Every local case runs three times at temperature 0 and counts only if all three agree.
3. The cheapest model that passes takes the task. Every candidate sits the exam. The bar is the best score any model reached, or one case from it; models under it drop out; the cheapest left wins and its score is committed as the snapshot. One task drawn out, then the pick for all seven:

   task	bar	cheapest at the bar	held-out	$ / 10k	resists injection	GPT-5.6-Sol, same task
   Email triage	9 of 11	gemma4:e2b-it-qat · local machine	10/11	≈0	0/2	9/11 · 5.72 (ledger)
   Expense fields	8 of 9	gemma4:e2b-it-qat · local machine	8/9	≈0	—	8/9 · 5.49 (ledger)
   Reply draft	9 of 10	gemma4:e2b-it-qat · local machine	10/10	≈0	1/2	10/10 · 16 (ledger)
   Digest of many items	7 of 10	gemma4:e4b-it-qat · local machine	7/10	≈0	—	8/10 · 44 (ledger)
   CRM brief over tool calls	15 of 18	gemma4-e2b-ctx16k · local machine	15/18	≈0	3/3	14/18 · 127 (ledger)
   CRM fields from a call	16 of 19	kimi-k3 · own VM, GPU node	17/19	92 (ledger)	—	15/19 · 65 (ledger)
   Call summary + action items	13 of 19	gpt-5.6-sol · vendor only	13/19	78 (ledger)	—	13/19 · 78 (ledger)

   Hosted rows are one pass; a one-case gap is noise, so the bar is one case wide. "Resists injection" counts the planted-instruction cases (two per task on email triage, reply draft and the CRM brief) the row did not obey; "—" means that model has not sat them yet. Dollars: tokens of that run at the 2026-10-08 ledger rate. The vendor flagship column is there for the reader's own subtraction; this page states no saving in advance.

4. A fine-tune ships only if it passes the same exam. When a small model almost passes, it is trained on the task's own training rows, on the machine it runs on, in minutes, and then sits the exam. If any held-out case that used to pass now fails, the version is refused, even when the total went up. Six attempts so far:

   candidate	held-out	what moved	verdict	cause named
   CRM follow-up, LoRA v1 · 11 training rows	12 → 12 of 18	2 up / 3 down · 3 held-out cases down, one pass	refused	nine of the eleven rows made one tool call; the adapter stops after one call
   CRM follow-up, LoRA v2 · 16 rows	12 → 12 of 18	2 up / 2 down · 2 stable held-out cases down, three loads	refused	flat total hiding two up, two down; too few rows with two calls in one turn
   CRM follow-up, LoRA v3 · 30 rows	12 → 11 of 18	4 up / 5 down · 3 stable held-out cases down, three loads	refused	no prompt mask, so the loss was 95% tool payload; it failed two of its own training rows
   CRM follow-up, LoRA v4 · 96 per-step examples, prompt masked	12 → 9 of 18	1 up / 7 down · 4 stable held-out cases down, three loads	refused	learned the decisions, and a “call again” reflex with them
   Request intake, thinking switched off by the chat template (no training)	19 → 17 of 20	0 up / 2 down · same speed as the adapter	refused	the switch loses the hand-over rules the adapter kept
   Request intake, LoRA · 22 rows, 9 minutes of training	19 → 18 of 20	0 up / 1 down · −87% output tokens, 4.2× faster; one pass, not snapshotted	held within one case; not promoted	removes the thinking channel; the one lost case is at N=20

   An average would have shipped v2 (flat total). The gate refused it. Every row is a probe folder under docs/probes/i0-*, i0b-*, i2*; the numbers here are parsed from those files.

Then the loop: every model release, runtime bump or price change re-sits the exam per case and the report lists what moved in both directions. A challenger replaces the incumbent only if it passes every stable case the incumbent passes. On 2 October a runtime upgrade moved 1 of 220 cases, 0 held-out.

Where the model runs

A score was taken on the runtime the row names. A build on your own card re-sits the exam before its row is claimed for it.

1. A 2B–4B model on a laptop holds every short decision task. It loses only long summaries.

One panel per task; no aggregate across tasks, because a sum hides the row a buyer cares about. Local marks sit in the ≈0 lane. The hosted rows are one pass each (2026-10-08 ledger); the local rows are three fresh loads that agreed on every case.

The honest limit. On call summaries with numbers the best model of any tier passes 14 of 19; the local one 10. That band is closed until real transcripts exist. Running the open weights on your own card is a residency choice, not a saving: at one request at a time an L4 costs more per task than the hosted per-token price on every row (call-fields on gemma-4-26b-a4b: hosted $2.14 per 10k (ledger) vs $5.47 on a RunPod L4 vs €8.82 on a Scaleway L4). modelled

2. Temperature zero is not determinism on hosted APIs. A gate that re-sits every case catches what moved.

Six same-day runs on 1 September (three with the creator endpoint pinned, three not), on that day's case sets. pass^k is the share of cases passed on all k runs — the τ-bench reliability metric. The gate rule: a case counts only when three fresh loads agree; a challenger replaces the incumbent only if it passes every stable case the incumbent passes.

3. Four fine-tunes of the local champion, four refusals, and the cause named.

The same gate that picks a model refuses a fine-tune. What the loop can show honestly on synthetic cases is a smaller model holding the champion's score and a format failure fixed; an accuracy gain needs cases that are not at ceiling, which means a client's.

Three times the instrument was wrong

The build ledger stays private; three of its entries are public as post-mortems, each in the same shape: what we believed, what the probe showed, what changed.

• The window, not the weights — three capability verdicts that were a 4,096-token truncation nobody had checked.
• The gate said ok while five verdicts moved — an aggregate regression check, blind to cancelling flips; what replaced it.
• A green gate that cannot go red — a test runner that reports "passed" by construction, and an answer key inside the instrument.

Reproduce it

git clone https://github.com/Michiel-DK/agent-exam-suite && cd agent-exam-suite
python3 docs/probes/pass3-column-2026-09-28/pass_k.py     # finding 2, from the committed snapshots and the six-run files
python3 docs/probes/matrix-2026-09-27/build_tables.py       # the seven tables (local wall times need a local results/ dir)
./.cline/test.sh                                            # the deterministic test gate, one process per file

Re-sitting an exam locally needs Ollama and the champion model: python3 sandbox/runner.py check email-triage compares a fresh run with the committed snapshot per case.