Skip to content
TEN Brief Ten verified stories a day 2026.11.10 KO

이 기사는 한국어로도 읽을 수 있습니다 →

Tech · 2 min read · Explainer

AI coding agent success rate — the best model fully finished only 26.4% of real tasks, versus 99% on older tests

In Co-Design Bench, a new benchmark from Alibaba and several Asian universities built from real user change requests to existing web apps, the best of ten models tested, OpenAI's GPT-6 Astra, met about 70% of requirements on average but fully completed only 26.4% of tasks. Claude Opus 5 and Qwen3.8-Max followed, and Grok 4.6 came last. Similar agents score above 99% on older, saturated benchmarks. All models did worse as requests combined more types of change, and rankings shifted depending on the agent harness used.

A developer seen from behind reviewing work on two monitors at a sunny desk

The three lines

  • Result — top model GPT-6 Astra fully completed 26.4%; about 70% of requirements met on average
  • Gap — older benchmarks show 99%+; the difference exceeds 35 points
  • Lesson — more simultaneous changes, more failures; harness choice changes rankings

Key questions

ai coding agent success rate
**On real change requests, the top model fully finished 26.4% of tasks. It met about 70% of requirements, but the rest needed human cleanup.** | Model (Co-Design Bench) | Result | |---|---| | GPT-6 Astra | 1st; 26.4% fully correct; ~70% requirements | | Claude Opus 5 | Near the top | | Qwen3.8-Max | Near the top | | Qwen3.8-27B | 2nd on large overhauls | | Grok 4.6 | Last of ten |
are ai coding benchmarks saturated
**Many are. Agents score above 99% on some, which says little about real-world usefulness.** | Benchmark | Scores | Limitation | |---|---|---| | WebCompass-edit and others | 99%+ | Saturated | | Co-Design Bench | 26.4% top (fully correct) | One fixed request per task; AI judges can err | | SWE-bench Verified vs real acceptance | Real acceptance ~24 pts lower (METR) | |
why do ai coding agents fail
**They struggle when a request mixes several kinds of change — layout, design, interaction and content — at once.** | Task type | GPT-6 Astra fully correct | |---|---| | Overall | 26.4% | | Multimodal inputs | 41% | | Repository-level tasks | Under 18% |

How often do AI coding agents finish what they are asked? On real user requests, even the best did so about one time in four. In Co-Design Bench, built by researchers from Alibaba and several Asian universities, OpenAI's GPT-6 Astra ranked first of ten models, meeting about 70% of requirements on average but fully completing only 26.4% of tasks. The same kind of agents score above 99% on older benchmarks.

1. What was tested

Instead of writing new programs, agents had to modify existing web apps according to change requests — sometimes delivered as images, audio, video or documents. Tasks came from real coding sessions, with a deliberate bias toward cases where users had to rephrase or roll back, meaning the AI had misunderstood at first. An AI judge tests each finished app in a browser, and a second judge rechecks every pass to catch mock-ups or hardcoded results. Only working features score.

2. Results

ModelResult
GPT-6 Astra (OpenAI)1st; 26.4% fully correct; ~70% of requirements
Claude Opus 5 (Anthropic)Near the top
Qwen3.8-Max (Alibaba)Near the top
Qwen3.8-27B (Alibaba)2nd on large overhauls
GLM-5.2~6x Astra's tokens on small changes, much lower score
Grok 4.6 (xAI)Last

Astra solved 41% of multimodal tasks but under 18% of repository-level ones. Every model declined as requests combined more change types. And the harness mattered: Qwen3.8-27B scored 75% or 56% on large overhauls depending on the environment, and rankings shifted.

3. How to read the numbers

BenchmarkScoresWhat it shows
Older tests (e.g. WebCompass-edit)99%+Saturated
Co-Design Bench26.4% topReal requests, finished end to end
Gap35+ pointsTest design drives scores
SWE-bench Verified vs real acceptance~24 points lower (METR)Benchmarks overstate field results

A 26.4% full-completion rate does not mean agents are useless — meeting about 70% of requirements is substantial help. It means human review cannot be skipped. The authors note limits: each task uses one fixed request with no follow-up questions, and AI judges can be wrong. Companies are also watching costs; see our report on Apple capping engineers' daily AI budgets, and on DeepSeek Harness, a cheaper agent framework.

Sources

  1. The Decoder — AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test (2026-11)
  2. arXiv — Wei et al., Co-Design Bench (arXiv:2610.35138)
  3. Co-Design Bench project page
  4. The Neuron — Everything That Happened in AI Today, Monday, November 9 (2026-11-09)

Verification

Published
Last modified
Cross-check
Checked against 4 independent sources.
Unverified
  • Total task count and per-model scores were not checked against the paper; figures follow The Decoder.
  • AI judges evaluate apps in a browser and may err, as the authors acknowledge.
  • The METR gap is a separate study, not a direct comparison with Co-Design Bench.
Authoring
Reviewed by a person before publication. The full process is described in the Editorial.

Ten stories, once each morning

We send the three-line summaries only; the full pieces stay on the site. One-click unsubscribe, any time.

Related