AI coding agent success rate — the best model fully finished only 26.4% of real tasks, versus 99% on older tests
In Co-Design Bench, a new benchmark from Alibaba and several Asian universities built from real user change requests to existing web apps, the best of ten models tested, OpenAI's GPT-6 Astra, met about 70% of requirements on average but fully completed only 26.4% of tasks. Claude Opus 5 and Qwen3.8-Max followed, and Grok 4.6 came last. Similar agents score above 99% on older, saturated benchmarks. All models did worse as requests combined more types of change, and rankings shifted depending on the agent harness used.
The three lines
- Result — top model GPT-6 Astra fully completed 26.4%; about 70% of requirements met on average
- Gap — older benchmarks show 99%+; the difference exceeds 35 points
- Lesson — more simultaneous changes, more failures; harness choice changes rankings
Key questions
- ai coding agent success rate
- **On real change requests, the top model fully finished 26.4% of tasks. It met about 70% of requirements, but the rest needed human cleanup.** | Model (Co-Design Bench) | Result | |---|---| | GPT-6 Astra | 1st; 26.4% fully correct; ~70% requirements | | Claude Opus 5 | Near the top | | Qwen3.8-Max | Near the top | | Qwen3.8-27B | 2nd on large overhauls | | Grok 4.6 | Last of ten |
- are ai coding benchmarks saturated
- **Many are. Agents score above 99% on some, which says little about real-world usefulness.** | Benchmark | Scores | Limitation | |---|---|---| | WebCompass-edit and others | 99%+ | Saturated | | Co-Design Bench | 26.4% top (fully correct) | One fixed request per task; AI judges can err | | SWE-bench Verified vs real acceptance | Real acceptance ~24 pts lower (METR) | |
- why do ai coding agents fail
- **They struggle when a request mixes several kinds of change — layout, design, interaction and content — at once.** | Task type | GPT-6 Astra fully correct | |---|---| | Overall | 26.4% | | Multimodal inputs | 41% | | Repository-level tasks | Under 18% |
How often do AI coding agents finish what they are asked? On real user requests, even the best did so about one time in four. In Co-Design Bench, built by researchers from Alibaba and several Asian universities, OpenAI's GPT-6 Astra ranked first of ten models, meeting about 70% of requirements on average but fully completing only 26.4% of tasks. The same kind of agents score above 99% on older benchmarks.
1. What was tested
Instead of writing new programs, agents had to modify existing web apps according to change requests — sometimes delivered as images, audio, video or documents. Tasks came from real coding sessions, with a deliberate bias toward cases where users had to rephrase or roll back, meaning the AI had misunderstood at first. An AI judge tests each finished app in a browser, and a second judge rechecks every pass to catch mock-ups or hardcoded results. Only working features score.
2. Results
| Model | Result |
|---|---|
| GPT-6 Astra (OpenAI) | 1st; 26.4% fully correct; ~70% of requirements |
| Claude Opus 5 (Anthropic) | Near the top |
| Qwen3.8-Max (Alibaba) | Near the top |
| Qwen3.8-27B (Alibaba) | 2nd on large overhauls |
| GLM-5.2 | ~6x Astra's tokens on small changes, much lower score |
| Grok 4.6 (xAI) | Last |
Astra solved 41% of multimodal tasks but under 18% of repository-level ones. Every model declined as requests combined more change types. And the harness mattered: Qwen3.8-27B scored 75% or 56% on large overhauls depending on the environment, and rankings shifted.
3. How to read the numbers
| Benchmark | Scores | What it shows |
|---|---|---|
| Older tests (e.g. WebCompass-edit) | 99%+ | Saturated |
| Co-Design Bench | 26.4% top | Real requests, finished end to end |
| Gap | 35+ points | Test design drives scores |
| SWE-bench Verified vs real acceptance | ~24 points lower (METR) | Benchmarks overstate field results |
A 26.4% full-completion rate does not mean agents are useless — meeting about 70% of requirements is substantial help. It means human review cannot be skipped. The authors note limits: each task uses one fixed request with no follow-up questions, and AI judges can be wrong. Companies are also watching costs; see our report on Apple capping engineers' daily AI budgets, and on DeepSeek Harness, a cheaper agent framework.