What is a distillation attack? Copying an AI by mining its answers — Anthropic says it cut the haul 90-99%
A distillation attack is when someone sends huge volumes of queries to a rival's AI model through its API, collects the answers and reasoning, and uses that data to train their own model — copying capability without paying for the research. The technique is the same as legitimate model distillation, where a company trains a small model on its own large one; the difference is permission. It typically breaches terms of service and can produce copies without the original's safety training. On October 12, 2026 Anthropic said new defenses had cut the data harvested by such attacks by about 90-99%, citing China's Zhipu (Z.ai) and its GLM-5.3 as an example.
The three lines
- Definition — mining a rival model's outputs via API to train a copy; same technique as distillation, without permission
- Risks — cheap clones that skip R&D, copies without safety training, leakage of sensitive skills like cyber offense
- Defense — Anthropic's preserved thinking and two-stage classifier cut harvested data ~90-99%; tens of thousands of accounts banned
Key questions
- Distillation attack meaning
- **Harvesting a rival model's answers at scale to train your own model, without permission.** | | Model distillation | Distillation attack | |---|---|---| | Teacher | Your own large model | Someone else's model | | Permission | Not needed | None — ToS violation | | Data | Generated internally | Mass API queries | | Safety | Designed in | Can be lost | | Goal | Cheaper, faster model | Copy capability, skip R&D |
- How does Anthropic stop distillation attacks
- **Three layers: preserved thinking, a two-stage classifier, and account bans.** | Measure | What it does | Reported effect | |---|---|---| | Preserved thinking | Accounts created on/after Aug 31, 2026 can't edit Claude's earlier context | Reasoning extraction 10-100x lower | | Two-stage classifier | Catches every method in Anthropic's September report | — | | Account bans | Tens of thousands linked to campaigns | — | | Overall | — | ~90-99% less data harvested |
- Distillation attack example Zhipu GLM
- **Anthropic says Zhipu tried to distill Claude's cyber skills while building GLM-5.3.** | Step | Anthropic's account | |---|---| | 1 | Zhipu targeted cyber capabilities in recent Claude models | | 2 | New safeguards degraded the attack | | 3 | It switched to an older Claude and another lab's model | | Zhipu's response | Not confirmed for this article |
You don't have to steal an AI model's weights to copy what it can do. Ask it hundreds of thousands of questions, collect the answers, and use them to train your own model. That is a distillation attack. On October 12, 2026 Anthropic said new defenses had cut the data such attacks harvest by about 90-99%. The same week, Anthropic reported $16.8 billion in Q3 revenue — and when capability is revenue, protecting it becomes core business.
1. Distillation vs. distillation attack
"Distillation" is a normal technique: a small "student" model learns to imitate a large "teacher", producing a cheaper, faster model.
| Legitimate distillation | Distillation attack | |
|---|---|---|
| Teacher | Your own large model | A competitor's model |
| Permission | Your own asset | None — most ToS forbid it |
| Data collection | Generated freely in-house | Mass API queries, spread across disguised accounts |
| Target | Lower cost | Skip years of R&D |
| Safety training | Designed together | Refusals and safeguards may not carry over |
| Example | Large → lightweight model | Anthropic's GLM-5.3 allegation against Zhipu |
Three problems follow. Research worth billions leaks for the price of API calls. A copy that learns the answers but not the safety training may answer dangerous requests the original refuses. And sensitive capabilities, such as cyber-offense skills, can walk out the door.
2. How Anthropic says it stopped it
| Measure | How it works | Reported effect |
|---|---|---|
| Preserved thinking | From Claude Fable 5.1 and Opus 5.5, API accounts created on or after Aug 31, 2026 cannot edit Claude's earlier context (its reasoning trace) | Reasoning extraction 10-100x lower |
| Two-stage classifier | Filter suspicious traffic, then judge precisely; catches every method in Anthropic's September report | — |
| Account bans | Tens of thousands of accounts tied to campaigns | — |
| Overall | — | ~90-99% less data harvested |
The first measure is the key one. Attackers want the model's reasoning, not just its answers, because reasoning traces teach a student far better. A common trick was editing earlier turns to steer the model into producing the desired reasoning; new accounts can no longer do that.
3. The defense has costs
| Cost | Detail |
|---|---|
| Lost revenue | Anthropic admits it turns away some paying traffic on purpose |
| Ordinary users | Context-editing limits and false positives can affect legitimate use |
| Displacement | Attackers move to older models or other labs (as Zhipu did, per Anthropic) |
| Arms race | Each published defense invites a new workaround |
Open-weight models need no distillation — their weights can simply be downloaded — so this cost falls on closed-model companies.
4. FAQ
| Question | Answer |
|---|---|
| Is it illegal? | Usually a terms-of-service breach; whether it amounts to trade-secret or copyright violation varies by jurisdiction and remains largely untested |
| Is saving chatbot answers distillation? | No — the issue is collecting them at scale to train a competing model |
| How is it detected? | Query patterns (rapid, systematic topic sweeps), linked accounts, attempts to manipulate reasoning traces |
| Do OpenAI and Google block it? | Their terms forbid training competing models on outputs; few publish defense metrics |
5. What remains open
- Independent verification: 90-99% is Anthropic's own measurement.
- Zhipu's side: not found.
- Legal status: whether courts will treat distillation as trade-secret misappropriation is unsettled; for a parallel fight over people rather than prompts, see "Apple v. OpenAI trade-secret case".
Sources
- BusinessWorld (Reuters) — Anthropic says new safeguards sharply reduce AI distillation attacks
- Technology.org — Anthropic Says New Safeguards Cut Claude Distillation by 90% to 99%
- Pulse 2.0 — Anthropic Says New Defenses Cut Data Harvested Through AI Distillation Attacks By 90% To 99%
- Times of AI — Anthropic Cuts AI Distillation Attacks by Up to 99%
- arXiv — What Does It Mean to Break a Distillation Defense?