Coding-agent benchmark: 99 tasks, 5 setups, 2 runs each
OpenRouter Auto Code, CodeRouter's default routing, and Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 on their own, on the same 99 real repository tasks. Each setup worked every task twice, in containers with no network access. What each one solved, what it cost, where our own router lost, and what we're changing next. Method and data are public so anyone can check them.
Disclosure: we build CodeRouter. OpenRouter Auto Code is a competing router; we called it through OpenRouter's public API, like any customer. Run on 2026-11-09 at that day's list prices. Write-up: OpenRouter Auto Code vs CodeRouter on 99 coding-agent tasks.
Results
A task counts as solved when the repository's tests pass after the agent's change. Intervals are 95%, from a bootstrap over tasks (both runs of a task resampled together).
| Setup | Solved | Completion | 95% interval | Cost / task | Customer-paid / task | Cost / solved task | Customer-paid / solved |
|---|---|---|---|---|---|---|---|
| All Claude Opus 5.5 | 130/198 | 65.7% | 56.6–74.7% | $0.247 | $0.247 | $0.376 | $0.376 |
| OpenRouter Auto Code | 125/198 | 63.1% | 54.0–72.2% | $0.246 | $0.260 | $0.390 | $0.412 |
| All Claude Sonnet 5.5 | 118/198 | 59.6% | 50.5–68.7% | $0.094 | $0.094 | $0.158 | $0.158 |
| CodeRouter default routing | 117/198 | 59.1% | 50.0–68.2% | $0.115 | $0.124 | $0.194 | $0.209 |
| All Claude Haiku 5.5 | 111/198 | 56.1% | 47.0–65.2% | $0.010 | $0.010 | $0.017 | $0.017 |
Cost = every request's input, cache-read, cache-write and output tokens × the model's list price, summed per session (Auto Code: the usage.cost OpenRouter returned for each request). Customer-paid: the Claude models direct = list price; Auto Code = list + OpenRouter's 5.5% fee on credit purchases; CodeRouter = list + 8% (pay as you go at the time of the run). Since November 11, 2026, CodeRouter bills list price with no markup plus a 5.5% fee on credit purchases, the same as Auto Code; the customer-paid figures here are as published, at the run's terms. Per-task cost intervals and medians are in the JSON.
Claude Fable 5.2 on 30 of these tasks: write-up and data.
The same 30 tasks with Fable 5.2 at Medium effort: write-up and data.
Paired comparisons
Same task, same run. “Only A / only B” counts the sessions where exactly one of the two solved the task; the p-value is an exact McNemar test on those. Differences in percentage points.
| A vs B | Completion A − B | 95% interval | p | Only A / only B | Cost A ÷ B | 95% interval | Cost / solved A ÷ B |
|---|---|---|---|---|---|---|---|
| Auto Code vs All-Opus | −2.5 | −8.6 to +3.5 | 0.46 | 12 / 17 | 1.00× | 0.91–1.10× | 1.04× |
| All-Sonnet vs All-Opus | −6.1 | −11.6 to −1.0 | 0.012 | 4 / 16 | 0.38× | 0.35–0.41× | 0.42× |
| CodeRouter vs All-Opus | −6.6 | −11.6 to −2.0 | 0.011 | 5 / 18 | 0.46× | 0.39–0.55× | 0.52× |
| All-Haiku vs All-Opus | −9.6 | −16.2 to −3.5 | 0.001 | 7 / 26 | 0.04× | 0.03–0.05× | 0.05× |
| CodeRouter vs Auto Code | −4.0 | −10.1 to +2.0 | 0.18 | 10 / 18 | 0.46× | 0.39–0.56× | 0.50× |
| CodeRouter vs All-Sonnet | −0.5 | −5.1 to +4.0 | 1.00 | 9 / 10 | 1.22× | 1.06–1.40× | 1.23× |
| Auto Code vs All-Sonnet | +3.5 | −2.0 to +9.1 | 0.23 | 16 / 9 | 2.62× | 2.37–2.91× | 2.47× |
| All-Sonnet vs All-Haiku | +3.5 | −2.5 to +10.1 | 0.30 | 20 / 13 | 9.68× | 8.60–10.77× | 9.11× |
Cost ratios at list price. Customer-paid ratios are in the JSON.
CodeRouter vs Auto Code
- Completion: 59.1% vs 63.1%, −4.0 points (95% interval −10.1 to +2.0, p = 0.18). The interval includes zero: on these 99 tasks neither one completes measurably more.
- Cost per task: 0.46× at list price (0.39–0.56×), 0.48× customer-paid.
- Cost per solved task: $0.194 vs $0.390 at list price (0.50×); $0.209 vs $0.412 customer-paid (0.51×).
- Auto Code vs all-Opus: completion −2.5 points (p = 0.46), cost 1.00× (0.91–1.10×) at list price, 1.05× customer-paid. OpenRouter reports Auto Code “in line with Claude Opus 5.5 while costing 74% less per task” on its own five benchmarks (announcement); in this setup we did not see the lower cost.
What we found about our own router
CodeRouter's default routing completed 6.6 points less than all-Opus (95% interval −11.6 to −2.0, p = 0.011), a significant difference, at 0.46× the cost. It did not beat all-Sonnet: completion −0.5 points (−5.1 to +4.0) at 1.22× the list cost (1.06–1.40×).
What the data shows: most sessions in our setup ran on Sonnet 5.5. 196 of 198 sessions started on Sonnet, and Sonnet served 1,834 of the requests. 24 sessions moved up to Opus (and 1 from Haiku to Sonnet), and those moves didn't add completions: on the same tasks and runs, those 24 sessions solved 9, while all-Sonnet solved 12 and all-Opus 13. Opus served 229 of 2,071 requests and cost $5.94 of the setup's $22.67 (26%).
What we're changing next: we keep optimizing our routing strategies and algorithms; these results are a snapshot of the version that ran on 2026-11-09. Next we're testing a revised routing policy on the same 99 tasks. Its rules, and how it will be compared with this run, were fixed before the run in a dated internal document. We will publish the result whatever it shows. Default routing is unchanged until then.
Where the money goes
| Setup | Cache-read share | Sessions that changed model | Model changes / session | Median steps |
|---|---|---|---|---|
| All Claude Opus 5.5 | 92% | 0 | 0.00 | 11 |
| OpenRouter Auto Code | 90% | 19 | 0.22 | 12 |
| All Claude Sonnet 5.5 | 90% | 0 | 0.00 | 9 |
| CodeRouter default routing | 89% | 25 | 0.13 | 8 |
| All Claude Haiku 5.5 | 92% | 0 | 0.00 | 12 |
Cache-read share = cache-read tokens ÷ all prompt tokens. Steps = model calls the agent made in a session.
Which models Auto Code and CodeRouter used
| Auto Code: model | Requests | Share | List cost | Sessions it started |
|---|---|---|---|---|
| GPT-6.1 Sol | 1,235 | 42% | $21.96 | 92 |
| Claude Opus 5.5 | 1,001 | 34% | $19.77 | 84 |
| Kimi K3.1 Pro | 386 | 13% | $5.43 | 14 |
| GLM-5.3 | 268 | 9% | $1.07 | 8 |
| Gemini 4 Argon | 28 | 1% | $0.56 | 0 |
| Total | 2,918 | 100% | $48.79 | 198 |
| CodeRouter: model | Requests | Share | List cost | Sessions it started |
|---|---|---|---|---|
| Claude Sonnet 5.5 | 1,834 | 89% | $16.73 | 196 |
| Claude Opus 5.5 | 229 | 11% | $5.94 | 0 |
| Claude Haiku 5.5 | 8 | 0% | $0.00 | 2 |
| Total | 2,071 | 100% | $22.67 | 198 |
“Sessions it started” = the model that served a session's first request. Auto Code's cache kept working through OpenRouter: 90% of its prompt tokens were cache reads, even though 19 of its sessions changed model.
Run-to-run variance
| Setup | Run 1 | Run 2 | Solved both | Solved one only | Solved neither | Run 1 vs 2, p |
|---|---|---|---|---|---|---|
| All Claude Opus 5.5 | 64 | 66 | 61 | 8 | 30 | 0.73 |
| OpenRouter Auto Code | 64 | 61 | 59 | 7 | 33 | 0.45 |
| All Claude Sonnet 5.5 | 60 | 58 | 55 | 8 | 36 | 0.73 |
| CodeRouter default routing | 55 | 62 | 53 | 11 | 35 | 0.07 |
| All Claude Haiku 5.5 | 56 | 55 | 48 | 15 | 36 | 1.00 |
Counts of tasks (of 99). One run of 99 tasks is not enough to rank these setups.
How we ran it
- Tasks: the SWE-rebench 2026_03 split (
nebius/SWE-rebench-leaderboard, CC BY 4.0, revision34d5a58864ac): real pull requests from Python repositories, created 2026-03-01 to 2026-05-12. 99 tasks from 70 repositories in the main analysis (see the checks below). Task ids are in the JSON. - Agent: mini-swe-agent 2.4.6 (bash tool only, benchmarks/swebench.yaml prompts unchanged) via LiteLLM 1.108.0. Only the model name (and the API endpoint) differs between setups. Limits per session: 150 steps, $2.50, 60 minutes, 60 s per command,
max_tokens8,192. - Setups: all-Opus, all-Sonnet and all-Haiku named the model on every request through CodeRouter's pass-through path (no routing, no fallback; cost is tokens × Anthropic list price either way). CodeRouter default =
model: autoon the production API, as any user gets it. Auto Code =openrouter/auto-codethrough OpenRouter's API. Auto Code chooses its own reasoning effort; every other setup used the model's default settings. - Prompt caching: mini-swe-agent's default for Anthropic models: a 5-minute cache marker on the last message of every turn, the same for every setup.
- Runs: Each setup ran every task twice on 2026-11-09. Run 2 started after run 1 ended, with the task order rotated, so the same setup reached the same task about an hour apart (no shared prompt cache). The five setups ran interleaved, 20 sessions at a time.
- Grading: swebench 5.0.2 harness (Docker) with the SWE-rebench evaluation spec: apply the patch, run the task's tests; resolved = the required failing tests pass and the passing tests still pass.
- Machine: Agent containers and grading ran on one rented cloud VM, used only to run containers. Nothing here measures the machine.
Validity checks
Reference fix and empty change
With the network off, each task's upstream fix was graded 3 times; 100 of the split's 110 tasks passed all 3. An empty change solved 0 of 110. Of the 108 tasks we ran, 9 no longer passed offline and are excluded from the main analysis:
| Task | Why it was dropped |
|---|---|
azure-samples__azure-search-openai-demo-3025 | its tests need network access; offline the test run does not start |
destine-climate-dt__aqua-2768 | some of its tests need network access (e.g. git clone from GitHub, data downloads, DNS); they fail offline even with the upstream fix |
geopython__pygeoapi-2338 | some of its tests need network access (e.g. git clone from GitHub, data downloads, DNS); they fail offline even with the upstream fix |
mempalace__mempalace-1004 | some of its tests need network access (e.g. git clone from GitHub, data downloads, DNS); they fail offline even with the upstream fix |
olofk__fusesoc-776_interface | some of its tests need network access (e.g. git clone from GitHub, data downloads, DNS); they fail offline even with the upstream fix |
pdm-project__pdm-3759 | some of its tests need network access (e.g. git clone from GitHub, data downloads, DNS); they fail offline even with the upstream fix |
pgmpy__pgmpy-3137 | some of its tests need network access (e.g. git clone from GitHub, data downloads, DNS); they fail offline even with the upstream fix |
reframe-hpc__reframe-3660_interface | some of its tests need network access (e.g. git clone from GitHub, data downloads, DNS); they fail offline even with the upstream fix |
zauberzeug__nicegui-5914 | a test that starts a local server failed in 1 of 3 offline reference-fix runs |
Sensitivity: re-grading 8 of them with the network-dependent tests excluded (107 tasks × 2 runs) gives All-Opus 66.4%, Auto Code 64.0%, All-Sonnet 59.8%, CodeRouter 59.8%, All-Haiku 55.6%. The order doesn't change.
Network isolation
Every packet from a container to outside the host was rejected and logged (Docker DOCKER-USER chain), for agent containers and grading containers alike. Verified before the run with a probe container: DNS, a GitHub IP on 443 and 1.1.1.1:53 were all unreachable. Images were pulled by the host's Docker daemon. 306,628 packets were blocked: 95,311 from agent containers and 211,299 from grading containers, almost all DNS lookups (port 53) made by tests and pip.
Commands that tried to fetch the task's own package or upstream source (flagged by a pattern search, each checked by hand in the session log). All were blocked:
| Setup | Run | Task | Command | Solved |
|---|---|---|---|---|
| All-Opus | 1 | psd-tools__psd-tools-593 | pip download psd-tools==1.14.2 | no |
| All-Opus | 2 | schemathesis__schemathesis-4087 | pip download schemathesis==99 | no |
| CodeRouter | 1 | psd-tools__psd-tools-593 | pip download psd-tools | no |
| CodeRouter | 2 | psd-tools__psd-tools-593 | pip download psd-tools | no |
| Auto Code | 1 | agronholm__anyio-1134 | pip download anyio==4.12.0 | yes (offline) |
| Auto Code | 2 | oemof__tespy-921 | urllib request for a source file on the project's GitHub dev branch | yes (offline) |
| Auto Code | 2 | pypa__twine-1309 | pip download twine==6.3.0 | no |
Memorization
0 of 990 submitted patches were identical to the upstream fix. The median overlap of changed lines with the upstream fix (Jaccard) was 0.08–0.12 per setup and run.
Submissions that weren't patches
Sonnet 5.5 sometimes submitted the output of a command like cat patch.txt | wc -l or | head -20 (a line count or the first lines) instead of the patch: 5 sessions in all-Sonnet and 5 in CodeRouter's setup, all served by Sonnet. None in the other setups. They count as not solved. Counting all of them as solved (an upper bound) would give all-Sonnet 62.1% and CodeRouter 61.6%, still below all-Opus. Grading errors: 0.
Limitations
- One agent harness (mini-swe-agent, bash tool only). Claude Code, Codex or Cursor sessions differ: other tools, prompts and context handling.
- 99 Python repository tasks (SWE-rebench, created March to May 2026). Claude Opus 5.5's published knowledge cutoff (June 2026) is later than the task dates, so we can't prove no model saw the fixes; no submitted patch was identical to the upstream fix.
- 2 runs per task. One setup's completion rate has a 95% interval of about ±7 (Wilson) to ±9 points (task bootstrap); paired differences between two setups have intervals of about ±5 to ±6 points, so smaller differences can't be told apart from noise.
- 9 of the 108 tasks we ran were dropped because their upstream fix doesn't pass offline (8 need network for some tests, 1 flaky). Re-grading 8 of them without the network-dependent tests doesn't change the order.
- List prices on 2026-11-09. Prices, models and routers change; results may change with model updates.
- OpenRouter Auto Code picks models per request and its routes change over time; this is one day's behavior.
- Limits: 150 steps, $2.50 and 60 minutes per session; max_tokens 8,192.
Data
- Summary JSON: every number on this page, the method, task ids, dropped tasks, checks and limitations.
- Per-session CSV (990 rows: task × setup × run): solved, list cost, steps, requests, input / cache-read / cache-write / output tokens, cache-read share, exit status. Which models served each setup is in the JSON, per setup.
Not published: the agent trajectories. The single-turn benchmark (176 single-file tasks) is on the benchmark page; it measures something different and its numbers don't carry over to agent sessions.