Disclosure: we build CodeRouter. OpenRouter Auto Code is a competing router; we called it through OpenRouter's public API, like any customer.
Update, Nov 12, 2026: since November 11, CodeRouter bills model list price with no markup, plus a 5.5% payment fee when you buy credit (a $20 top-up costs $21.10). The customer-paid figures below use list price +8%, CodeRouter's price on the run date; list-price figures are unaffected. Current terms: pricing.
On November 6, OpenRouter announced Auto Code, a router for coding agents, with this result: "Auto Code scores in line with Claude Opus 5.5 while costing 74% less per task", across five benchmarks including Deep SWE and Terminal Bench 4.0.
We ran Auto Code next to our own router and three single-model baselines on the same 99 real repository tasks, each task twice, in containers with no network access. In our setup, Auto Code's completion was close to Opus. So was its cost: 1.00× Opus per task.
This post is a build log: what we measured, where our router lost and why we think it did, and what we're changing next. The method and the data are public, so anyone can check them or rerun the comparison.
The results
99 tasks × 2 runs = 198 sessions per setup. A task counts as solved when the repository's tests pass after the agent's change. Intervals are 95%. Costs are at list price on November 9, 2026.
| Setup | Solved | Completion (95% interval) | Cost per task | Cost per solved task |
|---|---|---|---|---|
| All Claude Opus 5.5 | 130/198 | 65.7% (56.6–74.7%) | $0.247 | $0.376 |
| OpenRouter Auto Code | 125/198 | 63.1% (54.0–72.2%) | $0.246 | $0.390 |
| All Claude Sonnet 5.5 | 118/198 | 59.6% (50.5–68.7%) | $0.094 | $0.158 |
| CodeRouter default routing | 117/198 | 59.1% (50.0–68.2%) | $0.115 | $0.194 |
| All Claude Haiku 5.5 | 111/198 | 56.1% (47.0–65.2%) | $0.010 | $0.017 |
- All-Opus completed the most: 65.7%, at $0.247 per task.
- Auto Code: 63.1% at $0.246 per task, the same cost as Opus (1.00×, interval 0.91–1.10×). Its completion was within noise of Opus (−2.5 points, p = 0.46).
- CodeRouter: 59.1% at $0.115 per task (0.46× Opus). That is 6.6 points below Opus, and the difference is statistically significant (p = 0.011).
- All-Sonnet: 59.6% at $0.094 per task. CodeRouter matched its completion but cost 22% more. On this benchmark, our router did not beat simply using Sonnet.
- All-Haiku: 56.1% at $0.010 per task.
What customers pay is a little different: CodeRouter bills model list price +8%, and OpenRouter charges a 5.5% fee when you buy credits. With those added, CodeRouter is $0.124 per task ($0.209 per solved task) and Auto Code $0.260 ($0.412). The Claude-only rows are list price.
CodeRouter vs Auto Code
Paired on the same task and the same run:
- Completion: Auto Code solved 4.0 points more (63.1% vs 59.1%). The 95% interval runs from 10.1 points in Auto Code's favor to 2.0 points in ours (p = 0.18). On 99 tasks, we can't say either one completes more.
- Cost per task: CodeRouter cost 0.46× Auto Code at list price (interval 0.39–0.56×), 0.48× with each one's fees.
- Cost per solved task: $0.194 vs $0.390 at list price (0.50×), $0.209 vs $0.412 with fees (0.51×).
Why Auto Code cost what Opus costs: its model mix. It served its 2,918 requests with GPT-6.1 Sol (1,235 requests, $21.96), Claude Opus 5.5 (1,001 requests, $19.77), Kimi K3.1 Pro (386, $5.43), GLM-5.3 (268, $1.07) and Gemini 4 Argon (28, $0.56). The first request of a session went to GPT-6.1 Sol in 92 of 198 sessions and to Opus in 84. Prompt caching worked through OpenRouter: 90% of Auto Code's prompt tokens were cache reads, in line with the other setups (89–92%), even though 19 of its sessions changed model.
Why our cost result differs from OpenRouter's: a different harness, different tasks, a different day. OpenRouter's figure comes from its own five benchmarks; ours comes from one agent (mini-swe-agent, bash tool only) on Python repository tasks, and Auto Code's routes change over time. We're not saying OpenRouter's number is wrong on its benchmarks. We didn't see the lower cost on ours.
Where CodeRouter lost
Two results we don't like, both in the table:
- Against all-Opus: 6.6 points lower completion (interval −11.6 to −2.0, p = 0.011), at 0.46× the cost. So no "Opus quality for less" claim for coding-agent sessions: on this benchmark, it isn't true.
- Against all-Sonnet: the same completion (−0.5 points, interval −5.1 to +4.0) at 22% more list cost (1.22×, interval 1.06–1.40×).
What we see in the data: most sessions in our setup ran on Sonnet 5.5. 196 of 198 sessions started on Sonnet, and Sonnet served 1,834 of the 2,071 requests. 24 sessions moved up to Opus, and those moves didn't add completions: on the same tasks and runs, those 24 sessions solved 9, while all-Sonnet solved 12 and all-Opus 13. Opus served 229 requests but cost $5.94, about a quarter (26%) of the setup's spend. In short, we paid Opus prices in the sessions that moved up and didn't get more tasks done than Sonnet alone.
One more thing we found: Sonnet 5.5 sometimes submitted the output of cat patch.txt | wc -l (a line count) or | head -20 instead of the patch itself. That happened in 10 sessions, 5 in all-Sonnet and 5 in CodeRouter's (all served by Sonnet), and never in the other setups. They count as not solved. Even counting all 10 as solved, an upper bound, Sonnet would be at 62.1% and CodeRouter at 61.6%, still below Opus.
What we're changing next
We keep optimizing our routing strategies and algorithms; this post is a snapshot of the version that ran on November 9, 2026.
Next, we're testing a revised routing policy on the same 99 tasks. We fixed its rules before the run, in a dated internal document, along with how we'll compare it with this run. We'll publish the result whatever it shows. Until then, default routing doesn't change.
How we ran it
- Tasks: the SWE-rebench 2026_03 split (nebius/SWE-rebench-leaderboard, CC BY 4.0): real pull requests from Python repositories, created March 1 to May 12, 2026. 99 tasks from 70 repositories made the main analysis (see the checks below).
- Agent: mini-swe-agent 2.4.6 with its bash tool and its SWE-bench prompts unchanged. Only the model name (and the API endpoint) differs between setups. Limits per session: 150 steps, $2.50, 60 minutes, 60 seconds per command,
max_tokens8,192. - Setups: all-Opus, all-Sonnet and all-Haiku named the model on every request through CodeRouter's pass-through path (no routing, no fallback; cost is tokens × Anthropic list price either way). CodeRouter default =
model: autoon our production API, as any user gets it. Auto Code =openrouter/auto-codethrough OpenRouter's API, with the cost OpenRouter returned for each request. Auto Code chose its own reasoning effort; the other setups used each model's defaults. - Runs: every task twice per setup, about an hour apart, so no prompt cache was shared between runs. The five setups ran interleaved.
- Grading: the swebench 5.0.2 harness with SWE-rebench's evaluation spec.
- Cost: every request's input, cache-read, cache-write and output tokens × list price, summed per session.
- Machine: one rented cloud VM, used only to run the containers. Nothing here measures the machine.
Checks
- Offline: no container had network access, neither the agent's nor the grader's. 306,628 outbound packets were blocked, 95,311 from agent containers, almost all DNS lookups by tests and pip. Seven sessions ran a command to download the task's own package or upstream source; all seven were blocked.
- Reference fix: with the network off, a task's upstream fix had to pass 3 times out of 3, and an empty change had to fail. 9 of the 108 tasks we ran failed this (8 have tests that need the internet, 1 has a flaky local-server test), which leaves 99. Re-grading 8 of them without the network-dependent tests doesn't change the order: Opus 66.4%, Auto Code 64.0%, Sonnet 59.8%, CodeRouter 59.8%, Haiku 55.6%.
- Memorization: 0 of 990 submitted patches were identical to the upstream fix.
- Grading errors: none.
Limitations
- One agent harness, with a bash tool only. Claude Code, Codex or Cursor sessions use other tools and prompts, and results there can differ.
- 99 Python tasks, created March to May 2026. Claude Opus 5.5's published knowledge cutoff (June 2026) is after that, so we can't prove no model had seen the fixes.
- 2 runs per task. A paired difference needs to be about 5–6 points to stand out from noise here; one setup's completion interval is about ±9 points.
- 9 tasks we ran were dropped because their upstream fix doesn't pass offline.
- List prices on November 9, 2026. Models, prices and routers change; results may change with model updates.
- Auto Code picks models per request and its routes change over time. This is one day's behavior.
The data
The results page has every table, the paired comparisons, run-to-run variance, which models each router used, and each blocked fetch attempt. You can download the summary JSON and a per-session CSV (990 rows: task × setup × run, with solved, cost, steps and tokens). No agent trajectories.