Disclosure: we build CodeRouter, a router for coding agents. The open models in this test went through CodeRouter with each model named explicitly (no routing); Claude Sonnet 5.5 went to Anthropic's API directly.
Earlier today we showed that Claude Code runs on DeepSeek, GLM and Kimi, on one small bug fix. That said nothing about real work. So we gave Claude Code itself 30 real repository tasks on each model, graded the results with the tasks' own tests, and counted every token.
Claude Sonnet 5.5 solved the most, 16 of 30. GLM-5.3 came close with 14, at 4× Sonnet's cost per task. DeepSeek V4.1 Flash was the cheapest per solved task but solved 9. Kimi K2.6 also solved 9, at 4.8× Sonnet's cost per task, and ran into our 60-minute limit on half the tasks.
The results
| Model in Claude Code | Solved (of 30) | 95% interval | Cost per task | Cost per solved task | Median time per task |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 16 | 36–70% | $0.145 | $0.27 | 1.0 min |
| GLM-5.3 | 14 | 30–64% | $0.583 | $1.25 | 4.5 min |
| DeepSeek V4.1 Flash | 9 | 17–48% | $0.033 | $0.11 | 4.0 min |
| Kimi K2.6 | 9 | 17–48% | $0.703 | $2.34 | 54 min |
Costs are list prices for the tokens each request reported, cache reads and writes included, with no fees. Same 30 tasks, same prompt, same limits, one run per model.
Against Sonnet on the same tasks (exact McNemar test): DeepSeek solved fewer (8 tasks only Sonnet solved, 1 only DeepSeek; p = 0.04), Kimi likely fewer (9 vs 2; p = 0.07), and GLM-5.3 was not distinguishable (3 vs 1; p = 0.63). With 30 tasks and one run, a 2-task gap is noise; a 7-task gap probably isn't.
Every model solved at least one task Sonnet didn't, but only one or two each. 12 tasks were solved by none of the four.
For context, our minimal agent harness (mini-swe-agent, one bash tool) solved 17 of the same 30 tasks with Sonnet 5.5 at $0.082 per task (published here). Claude Code solved one fewer at 1.8× the cost per task, and finished a typical task in a minute.
What happened with each model
Claude Sonnet 5.5. A median of 8 model requests per task, and the shortest sessions of the four: Claude Code is built around Claude models, and it shows. Its cost per solved task, $0.27, was second only to DeepSeek's.
GLM-5.3. Close to Sonnet on results, but it took a median of 31 requests per task where Sonnet took 8, and Claude Code resends the whole conversation each time. About 97% of GLM's input was cache reads, and GLM-5.3's cache-read price ($0.308 per million) is 3× Sonnet 5.5's ($0.10). More steps at a higher cache price is how a cheaper-looking model ends up at 4× the cost per task.
DeepSeek V4.1 Flash. $0.033 per task, about a quarter of Sonnet's, and $0.11 per solved task, the lowest. Its cache-read price is $0.006 per million, so long sessions barely cost anything. It solved 7 fewer tasks than Sonnet, and 3 sessions ran into Claude Code's 150-turn limit.
Kimi K2.6. 15 of its 30 sessions hit our 60-minute limit, and 7 ended with no code change. A median task took 54 minutes, almost all of it spent waiting for the model. Kimi was billed a median of 60,000 output tokens per task, against 3,600 for Sonnet, and we estimate about 87% of them never appeared in what Claude Code received: reasoning, not code or tool calls. (Sonnet's thinking is hidden too; by the same estimate about half of its output.) One caveat: Kimi reached Claude Code through CodeRouter, which runs K2.6 with thinking on for agent turns (Moonshot's default) and, because a client like Claude Code can't return the earlier turns' reasoning, sends it back empty. K2.6's default setting ignores earlier turns' reasoning anyway, so we don't expect that to change much, but we haven't measured Kimi through Moonshot's own Anthropic-compatible endpoint yet. We will, and we'll publish what it shows.
What this means if you use Claude Code
- Sonnet 5.5 is still the default to beat in Claude Code. It solved the most, fastest, and its cost per solved task was the second lowest.
- An open model isn't automatically cheaper in Claude Code. Most of the input is cache reads, so the cache-read price and the number of steps matter more than the input price on the list. Compare cost per solved task, not price per token.
- DeepSeek V4.1 Flash is the budget option: under a quarter of Sonnet's cost per task, with fewer tasks solved. A fit for routine work you'll review anyway.
- GLM-5.3 gets close to Sonnet on results, without being cheaper.
- Kimi K2.6 at Claude Code's defaults was slow and expensive in this test. Moonshot also plans to retire the K2 series at the end of December 2026.
These results come from one run on 30 Python tasks; your mix of tasks may differ. To try a model on your own work, the setup is in our Claude Code guide.
How we ran it
- Tasks: 30 tasks drawn at random (seed 20261110) from the 99 SWE-rebench tasks in our coding-agent benchmark: Python pull requests created between March 1 and May 12, 2026 (nebius/SWE-rebench-leaderboard, CC BY 4.0). Each task's official fix passed and an empty change failed on our grading machine (30 of 30 and 0 of 30).
- Agent: Claude Code 2.1.282 in headless mode, as shipped: its default system prompt, tools and thinking settings. Every model variable (main, Opus, Sonnet, Haiku and subagent) set to the model under test, so every request went to that model. The prompt gave the issue text and asked for a fix to the non-test source files, without changing tests or configuration and without committing.
- Limits: 150 turns, 60 minutes and $3 per task (the cost limit stopped one request in one GLM session).
- Isolation: each task ran in its own container with no network access except a relay to our metering proxy. WebSearch and WebFetch were disabled. Blocked network attempts were logged: DNS lookups only, none reached the internet.
- Grading: the official swebench harness (5.0.2). Solved means the task's failing tests now pass and its passing tests still pass.
- Cost: tokens from each response's usage × list price. Open models: CodeRouter's model list on November 12, the provider's list price with no markup (DeepSeek at its peak-hour price). Sonnet 5.5: Anthropic's list price.
- A rerun: our GLM account at Zhipu ran out of balance during the run, which cut 15 GLM-5.3 sessions short. We discarded them and reran those 15 tasks from scratch after topping up; only the reruns count.
Limitations
- One run per model and 30 tasks: intervals are wide, and a rerun could move each model by a few tasks.
- Claude Code is tuned for Claude models; its prompts and tools may favour Sonnet. That is the point of the test: how these models do inside Claude Code, not in general.
- Kimi K2.6 hit the 60-minute limit on half the tasks; a longer limit would raise its cost and maybe its score.
- The open models went through CodeRouter, which translates between Anthropic's format and each provider's (see the Kimi caveat above).
- Python repositories only.
The data
Per-session results (task, model, solved, how the session ended, turns, time, requests, input, cache-read, cache-write and output tokens, cost, patch size) and the aggregates, with method notes:
- coderouter-claude-code-open-models-2026-11.json
- coderouter-claude-code-open-models-2026-11-sessions.csv