Back to Blog

Claude Code on Open Models, 30 Real Tasks: Kimi K2.6 Cost 4.8× Sonnet 5.5 per Task and Solved 7 Fewer

2026-11-12·7 min read·CodeRouter Team

Disclosure: we build CodeRouter, a router for coding agents. The open models in this test went through CodeRouter with each model named explicitly (no routing); Claude Sonnet 5.5 went to Anthropic's API directly.

Earlier today we showed that Claude Code runs on DeepSeek, GLM and Kimi, on one small bug fix. That said nothing about real work. So we gave Claude Code itself 30 real repository tasks on each model, graded the results with the tasks' own tests, and counted every token.

Claude Sonnet 5.5 solved the most, 16 of 30. GLM-5.3 came close with 14, at 4× Sonnet's cost per task. DeepSeek V4.1 Flash was the cheapest per solved task but solved 9. Kimi K2.6 also solved 9, at 4.8× Sonnet's cost per task, and ran into our 60-minute limit on half the tasks.

The results

Model in Claude Code Solved (of 30) 95% interval Cost per task Cost per solved task Median time per task
Claude Sonnet 5.5 16 36–70% $0.145 $0.27 1.0 min
GLM-5.3 14 30–64% $0.583 $1.25 4.5 min
DeepSeek V4.1 Flash 9 17–48% $0.033 $0.11 4.0 min
Kimi K2.6 9 17–48% $0.703 $2.34 54 min

Costs are list prices for the tokens each request reported, cache reads and writes included, with no fees. Same 30 tasks, same prompt, same limits, one run per model.

Against Sonnet on the same tasks (exact McNemar test): DeepSeek solved fewer (8 tasks only Sonnet solved, 1 only DeepSeek; p = 0.04), Kimi likely fewer (9 vs 2; p = 0.07), and GLM-5.3 was not distinguishable (3 vs 1; p = 0.63). With 30 tasks and one run, a 2-task gap is noise; a 7-task gap probably isn't.

Every model solved at least one task Sonnet didn't, but only one or two each. 12 tasks were solved by none of the four.

For context, our minimal agent harness (mini-swe-agent, one bash tool) solved 17 of the same 30 tasks with Sonnet 5.5 at $0.082 per task (published here). Claude Code solved one fewer at 1.8× the cost per task, and finished a typical task in a minute.

What happened with each model

Claude Sonnet 5.5. A median of 8 model requests per task, and the shortest sessions of the four: Claude Code is built around Claude models, and it shows. Its cost per solved task, $0.27, was second only to DeepSeek's.

GLM-5.3. Close to Sonnet on results, but it took a median of 31 requests per task where Sonnet took 8, and Claude Code resends the whole conversation each time. About 97% of GLM's input was cache reads, and GLM-5.3's cache-read price ($0.308 per million) is 3× Sonnet 5.5's ($0.10). More steps at a higher cache price is how a cheaper-looking model ends up at 4× the cost per task.

DeepSeek V4.1 Flash. $0.033 per task, about a quarter of Sonnet's, and $0.11 per solved task, the lowest. Its cache-read price is $0.006 per million, so long sessions barely cost anything. It solved 7 fewer tasks than Sonnet, and 3 sessions ran into Claude Code's 150-turn limit.

Kimi K2.6. 15 of its 30 sessions hit our 60-minute limit, and 7 ended with no code change. A median task took 54 minutes, almost all of it spent waiting for the model. Kimi was billed a median of 60,000 output tokens per task, against 3,600 for Sonnet, and we estimate about 87% of them never appeared in what Claude Code received: reasoning, not code or tool calls. (Sonnet's thinking is hidden too; by the same estimate about half of its output.) One caveat: Kimi reached Claude Code through CodeRouter, which runs K2.6 with thinking on for agent turns (Moonshot's default) and, because a client like Claude Code can't return the earlier turns' reasoning, sends it back empty. K2.6's default setting ignores earlier turns' reasoning anyway, so we don't expect that to change much, but we haven't measured Kimi through Moonshot's own Anthropic-compatible endpoint yet. We will, and we'll publish what it shows.

What this means if you use Claude Code

These results come from one run on 30 Python tasks; your mix of tasks may differ. To try a model on your own work, the setup is in our Claude Code guide.

How we ran it

Limitations

The data

Per-session results (task, model, solved, how the session ended, turns, time, requests, input, cache-read, cache-write and output tokens, cost, patch size) and the aggregates, with method notes:

Related reading

Try phase-aware routing on your own workload

One base URL for Claude Code, Cursor or Aider. Free trial, no credit card.

Start free trial

Get weekly AI cost optimization tips

Join developers cutting their agent bills