What a coding benchmark is — why one model scores 28% and 84% at once
A coding benchmark scores an AI model on a fixed set of software tasks and reports a success rate. The same model routinely posts figures as far apart as 28% and 84%, because each benchmark measures a different capability under different scoring rules, and most published scores are reported by the model's own developer
The three lines
- What they are — fixed sets of software tasks given to a model, scored as a success rate; families include SWE-bench-style issue resolution, terminal multi-step work, and security benchmarks
- Why scores diverge — task difficulty, number of attempts allowed, tool access, and automated scoring method all differ between benchmarks
- How to read one — check who measured it, how many attempts, what tools and scaffolding, and whether the benchmark version matches
Key questions
- What is a coding benchmark?
- A standardized test of an AI model's programming ability. Tasks are drawn from real open-source repository issues, or multi-step terminal work, or security vulnerability discovery, and the model's success rate is reported as a percentage. Well-known examples include the SWE-bench family, Terminal-Bench and CyberGym.
- Why do scores differ so much for one model?
- Because they measure different things. GLM-5.3 scored 84.5% on CyberGym and 28.3% on Terminal-Bench 3.0 — a threefold spread, and no contradiction. One is a security task success rate; the other is multi-step terminal work. They are different exams with different difficulty and scoring.
- Can benchmark scores be trusted?
- Check who published them first. Scores that appear alongside a model launch are almost always measured and reported by the developer. The same model scores very differently depending on attempts allowed, tools available and prompt construction, so it is safest to read them as claims until a third party reproduces them under stated conditions.
Every new AI model arrives with a table. The benchmark table.
Look closely and something strange happens. The same model scores 84.5% on one row and 28.3% on another. A threefold gap.
Both are true. Here is why.
1. What a coding benchmark does
The principle is a school exam.
- Fix a set of problems in advance
- Give them to the model
- Judge correctness automatically
- Report the success rate as a percentage
The difference is in step 3. Coding benchmarks mostly grade by running tests. The model's patch is executed; if the test suite passes, it counts. No human is scoring code quality.
2. What differs by family
| Family | Examples | Measures | Difficulty character |
|---|---|---|---|
| Real issue resolution | SWE-bench, DeepSWE | Actual bugs and feature requests in open-source repos | Requires codebase comprehension |
| Terminal multi-step | Terminal-Bench | Completing multi-stage command-line work | Requires tool use and state tracking |
| Security | CyberGym, ExploitBench | Vulnerability discovery, proof-of-concept code | Narrow, patterned domain |
| Automation | AutomationBench | Automating repetitive work | Task definitions relatively clear |
This is where the threefold gap comes from. Security tasks are comparatively patterned, so high scores come more easily; multi-step terminal work fails entirely if one stage goes wrong, so scores run low.
3. A worked example — GLM-5.3 (released August 14, 2026)
| Benchmark | Score |
|---|---|
| CyberGym | 84.5% |
| DeepSWE 1.1 | 66.9% |
| ExploitBench | 54.4% |
| AutomationBench | 48.2% |
| Terminal-Bench 3.0 | 28.3% |
These five rows do not contradict each other. They are five different exams.
The common misreading follows a predictable path. Marketing leads with 84.5%, an article carries that number alone, and a reader concludes the model "is 84% good." The 28.3% at the bottom of the same table disappears.
4. Four conditions that swing a score
The same model on the same benchmark can post different numbers, because the conditions differed.
| Condition | Why it matters |
|---|---|
| Attempts | Getting it right first try (pass@1) and getting it right in any of k tries (pass@k) are entirely different numbers |
| Tool access | Whether code execution, file search or web access was allowed changes results substantially |
| Scaffolding | The performance of the agent framework wrapping the model gets folded into the score |
| Version | Terminal-Bench 3.0 and 2.0 are different exams; different versions are not comparable |
Before comparing two reported scores, check that all four match. If any one differs, the comparison does not hold.
5. The deeper structural problem — contamination
Benchmarks carry one fundamental weakness: the problems and their solutions are on the public internet.
If a benchmark draws its tasks from open-source repository issues, that repository's code and its fix commits may already sit in the training data. In that case the model is closer to recalling the answer than solving the problem.
This is data contamination. Countermeasures exist:
- Using only issues dated after the training cutoff
- Maintaining private holdout sets
- Perturbing problems to test for memorization
But verifying contamination for a specific model from the outside is usually impossible, because training data is not disclosed.
6. A checklist for reading a score
Four questions when a new benchmark table appears.
| # | Question | If you cannot answer it |
|---|---|---|
| 1 | Who measured it — the developer or a third party? | Read it as a claim |
| 2 | Attempts — pass@1 or pass@k? | Do not compare it to another model |
| 3 | What tools and scaffolding? | Do not attribute the gap to the model |
| 4 | Same benchmark version? | Do not build a comparison table |
For scores published at launch, question 1 answers itself: the developer. That is why this page writes "vendor-reported" every time it covers GLM-5.3, DeepSeek V4-Pro or Gemini 3.7 Flash.
7. Common questions
Q. Does 70% on SWE-bench mean replacing 70% of developers? No. SWE-bench measures the success rate of fixing well-specified issues in code. It does not measure deciding what to build, asking clarifying questions when requirements are vague, system design, or working with people.
Q. Do higher scores mean better real-world use? Correlated, but not proportional. No published study quantifying that relationship was located for this article.
Q. Which benchmark should I look at? The one closest to your use. Modifying existing codebases points to the SWE-bench family; command-line automation points to Terminal-Bench.
Q. What happens when a benchmark hits 100%? It is effectively finished. Saturated scores lose discriminating power and a harder successor version appears — which is why benchmarks carry version numbers.
8. What remains unverified
The cited scores are vendor-reported. The GLM-5.3 figures here have not been through third-party verification.
Contamination is mostly unverifiable. While training data stays private, outside adjudication is limited.
The link to real output is unquantified. No published research quantifying how benchmark scores translate into engineering productivity was located for this article.
Related coverage: "What a frontier model is" and "GLM-5.3 claims a 50% coding jump."
Sources
- aireleasetracker — GLM-5.3: Benchmarks, Specs & Release Date
- Qubrid AI — GLM-5.3 Benchmarks, Architecture & Pricing (Aug 2026)
- Memeburn — GLM 5.3 Is Here: Benchmarks, Pricing, Coding and What's New
- atoms.dev — GLM-5.3 Complete Guide: Benchmarks, API, Coding, and Open Weights
- llm-stats — AI Updates (August 2026): Latest AI Model Releases