Back to Blog

OpenRouter Auto Code vs CodeRouter on 99 Coding-Agent Tasks: Similar Completion, Half the Cost per Solved Task

2026-11-10·8 min read·CodeRouter Team

Disclosure: we build CodeRouter. OpenRouter Auto Code is a competing router; we called it through OpenRouter's public API, like any customer.

Update, Nov 12, 2026: since November 11, CodeRouter bills model list price with no markup, plus a 5.5% payment fee when you buy credit (a $20 top-up costs $21.10). The customer-paid figures below use list price +8%, CodeRouter's price on the run date; list-price figures are unaffected. Current terms: pricing.

On November 6, OpenRouter announced Auto Code, a router for coding agents, with this result: "Auto Code scores in line with Claude Opus 5.5 while costing 74% less per task", across five benchmarks including Deep SWE and Terminal Bench 4.0.

We ran Auto Code next to our own router and three single-model baselines on the same 99 real repository tasks, each task twice, in containers with no network access. In our setup, Auto Code's completion was close to Opus. So was its cost: 1.00× Opus per task.

This post is a build log: what we measured, where our router lost and why we think it did, and what we're changing next. The method and the data are public, so anyone can check them or rerun the comparison.

The results

99 tasks × 2 runs = 198 sessions per setup. A task counts as solved when the repository's tests pass after the agent's change. Intervals are 95%. Costs are at list price on November 9, 2026.

Setup Solved Completion (95% interval) Cost per task Cost per solved task
All Claude Opus 5.5 130/198 65.7% (56.6–74.7%) $0.247 $0.376
OpenRouter Auto Code 125/198 63.1% (54.0–72.2%) $0.246 $0.390
All Claude Sonnet 5.5 118/198 59.6% (50.5–68.7%) $0.094 $0.158
CodeRouter default routing 117/198 59.1% (50.0–68.2%) $0.115 $0.194
All Claude Haiku 5.5 111/198 56.1% (47.0–65.2%) $0.010 $0.017

What customers pay is a little different: CodeRouter bills model list price +8%, and OpenRouter charges a 5.5% fee when you buy credits. With those added, CodeRouter is $0.124 per task ($0.209 per solved task) and Auto Code $0.260 ($0.412). The Claude-only rows are list price.

CodeRouter vs Auto Code

Paired on the same task and the same run:

Why Auto Code cost what Opus costs: its model mix. It served its 2,918 requests with GPT-6.1 Sol (1,235 requests, $21.96), Claude Opus 5.5 (1,001 requests, $19.77), Kimi K3.1 Pro (386, $5.43), GLM-5.3 (268, $1.07) and Gemini 4 Argon (28, $0.56). The first request of a session went to GPT-6.1 Sol in 92 of 198 sessions and to Opus in 84. Prompt caching worked through OpenRouter: 90% of Auto Code's prompt tokens were cache reads, in line with the other setups (89–92%), even though 19 of its sessions changed model.

Why our cost result differs from OpenRouter's: a different harness, different tasks, a different day. OpenRouter's figure comes from its own five benchmarks; ours comes from one agent (mini-swe-agent, bash tool only) on Python repository tasks, and Auto Code's routes change over time. We're not saying OpenRouter's number is wrong on its benchmarks. We didn't see the lower cost on ours.

Where CodeRouter lost

Two results we don't like, both in the table:

  1. Against all-Opus: 6.6 points lower completion (interval −11.6 to −2.0, p = 0.011), at 0.46× the cost. So no "Opus quality for less" claim for coding-agent sessions: on this benchmark, it isn't true.
  2. Against all-Sonnet: the same completion (−0.5 points, interval −5.1 to +4.0) at 22% more list cost (1.22×, interval 1.06–1.40×).

What we see in the data: most sessions in our setup ran on Sonnet 5.5. 196 of 198 sessions started on Sonnet, and Sonnet served 1,834 of the 2,071 requests. 24 sessions moved up to Opus, and those moves didn't add completions: on the same tasks and runs, those 24 sessions solved 9, while all-Sonnet solved 12 and all-Opus 13. Opus served 229 requests but cost $5.94, about a quarter (26%) of the setup's spend. In short, we paid Opus prices in the sessions that moved up and didn't get more tasks done than Sonnet alone.

One more thing we found: Sonnet 5.5 sometimes submitted the output of cat patch.txt | wc -l (a line count) or | head -20 instead of the patch itself. That happened in 10 sessions, 5 in all-Sonnet and 5 in CodeRouter's (all served by Sonnet), and never in the other setups. They count as not solved. Even counting all 10 as solved, an upper bound, Sonnet would be at 62.1% and CodeRouter at 61.6%, still below Opus.

What we're changing next

We keep optimizing our routing strategies and algorithms; this post is a snapshot of the version that ran on November 9, 2026.

Next, we're testing a revised routing policy on the same 99 tasks. We fixed its rules before the run, in a dated internal document, along with how we'll compare it with this run. We'll publish the result whatever it shows. Until then, default routing doesn't change.

How we ran it

Checks

Limitations

The data

The results page has every table, the paired comparisons, run-to-run variance, which models each router used, and each blocked fetch attempt. You can download the summary JSON and a per-session CSV (990 rows: task × setup × run, with solved, cost, steps and tokens). No agent trajectories.

Try phase-aware routing on your own workload

One base URL for Claude Code, Cursor or Aider. Free trial, no credit card.

Start free trial

Get weekly AI cost optimization tips

Join developers cutting their agent bills