Benchmark methodology and results
In TokenWatt's 176-task coding benchmark, routed requests solved 96–100% of tasks for 56–70% less than sending every request to Opus, across runs. The current router, routing v3, solved 176 of 176 in its first run for $0.77 — 30% of the all-Opus cost of $2.56.
Who measured this. TokenWatt is a measurement project from the same team as CodeRouter, so this is not an independent benchmark. That is why the method, every aggregate and the limits are on this page, and why the numbers are downloadable as JSON.
Method
- Tasks. 176 single-file Python tasks: Exercism exercises plus the hard set of Aider's polyglot benchmark.
- One shot, one retry. The model sees the instructions and the stub file and returns the whole file. The task's tests run. If they fail, the model gets one retry with the failing test output.
- Scoring. "First try" means the tests passed without the retry; "solved" or "with one retry" means they passed after at most one retry.
- Cost. Provider list prices as checked on 2026-11-05 (DeepSeek at its peak-hour price), applied to each request's logged tokens, including the retry.
- Baseline. Every request sent to Claude Opus 5.5: 176/176 solved with the retry, for $2.56. "56–70% less" is measured against this.
Results by router
| Run | When | Solved | First try | Cost | vs all-Opus |
|---|---|---|---|---|---|
| All-Opus baseline | — | 176/176 | — | $2.56 | 100% |
| Earlier router, 3 runs | October 2026 | 96.0%–97.7% | — | 41–44% of all-Opus | 56–59% less |
| Routing v3, first run | 2026-11-04 | 176/176 | 88.6% | $0.77 | 30% (70% less) |
Routing v3 works like this: for each request a small judgment model (Jev, from TypeSafe) reads a short excerpt — the latest message and the latest tool output — and judges the coding phase, how hard the task is and whether the last turn failed. Code then picks the tier: Claude Haiku 5.5 (alternate DeepSeek Chat), Kimi K2.6, Claude Sonnet 5.5, Claude Opus 5.5. It starts on the cheapest tier that can handle the task and moves up one tier only when tests or errors show a failure.
Every model on the same tasks
On 2026-11-03, each model was called directly through CodeRouter's production path on the same tasks, one run per model. 175 tasks are used for the matrix comparison because one Kimi K3.1 Pro request errored.
| Model | First try | With one retry | Total cost (incl. retry) |
|---|---|---|---|
| Claude Opus 5.5 | 98.3% | 100.0% | $2.61 |
| Claude Sonnet 5.5 | 93.7% | 100.0% | $1.48 |
| Kimi K2.6 | 86.9% | 94.9% | $0.57 |
| Claude Haiku 5.5 | 72.0% | 93.7% | $0.15 |
| DeepSeek Chat | 70.3% | 80.6% | $0.11 |
| DeepSeek V4 Pro | 68.6% | 78.9% | $0.36 |
| Kimi K3.1 Pro | 67.4% | 90.9% | $1.67 |
| GLM-5.3 | 62.3% | 89.1% | $0.99 |
Two things stand out. The retry matters most for the cheaper models — Claude Haiku 5.5 goes from 72.0% to 93.7% — which is why routing starts cheap and escalates on a failure. And price doesn't predict quality on these tasks: some of the more expensive models solved fewer tasks than cheaper ones.
Routing compared with simpler strategies
| Strategy | Solved | Cost | Tasks |
|---|---|---|---|
| Opus 5.5 for everything | 100.0% | $2.61 | 175 (matrix) |
| Sonnet 5.5 for everything | 100.0% | $1.48 | 175 (matrix) |
| Routing v3, first run | 100.0% | $0.77 | 176 (separate run) |
| Cheapest-first cascade: DeepSeek Chat, then Kimi K2.6, then Claude Sonnet 5.5; escalate on test failure. | 97.1% | $0.49 | 175 |
| Oracle: The cheapest model that passes each task, chosen after the fact. A lower bound, not something a live router can do. | 99.4% | $0.29 | 175 |
On these tasks a plain cheapest-first cascade was cheaper than routing v3 ($0.49 vs $0.77) and solved slightly fewer (97.1% vs 100.0%). The benchmark suits a cascade: every task has tests that say exactly when to escalate, and turns are short, so starting over on another model costs little. In a long coding-agent session, switching models loses the prompt cache — which is why routing v3 keeps a session on one model, and needs two failures before it moves up when the context is 30k tokens or more. This benchmark doesn't measure that.
The oracle picks, with hindsight, the cheapest model that passes each task. No live router can do that; it shows how much room is left.
Limitations
- Single-file Python tasks only.
- Public tasks: models may have seen them during training.
- One run per model in the matrix, and one run of routing v3.
- Single-turn, short contexts with no prompt-cache cost — unlike long coding-agent sessions.
- Measured by TokenWatt, which is run by the same team as CodeRouter.
Your own savings depend on your mix of planning, implementation, debugging and edits, and on your cache hit rate. For single requests, the CodeRouter dashboard shows your cost next to an estimate of the same tokens at Opus list price. Coding-agent sessions get no savings figure: a stronger model takes a different number of steps, so the same tokens at Opus price aren't what an all-Opus session would cost. For real per-developer usage rather than benchmark tasks, see our team's numbers on the Claude Code pricing page. Those show what our team spent at list prices, not what the same work would cost on one model: unlike the all-Opus baseline here, which is a real run, repricing everyday traffic on one model can't tell you that, because a different model takes a different number of steps.
Dates and data
- Earlier router, three runs: October 2026.
- Model matrix: 2026-11-03.
- Routing v3, first run: 2026-11-04.
- Aggregates as JSON (no per-task data): https://www.coderouter.io/benchmarks/coderouter-176-2026-11.json. The same file carries our team's per-developer spend at list prices under
team_usage_2026_05_09(corrected 2026-11-07; see itscorrectednote).