What an AI safety institute is — the government testers who get AI models before you do
An AI safety institute is a government agency that evaluates the most advanced AI models for severe risks such as cyberattacks, biological or chemical weapons uplift and loss of control, ideally by testing them before public release. The UK created the first in November 2023 and renamed it the AI Security Institute in February 2025. The US version at NIST was rebranded the Center for AI Standards and Innovation (CAISI) in June 2025, and South Korea opened its AI Safety Institute under ETRI on November 27, 2024. Because institutes get unreleased models through voluntary deals rather than legal mandates, a 17-day US freeze on UK access in September-October 2026 meant Anthropic's Claude Mythos 5.1 skipped British testing.
The three lines
- Definition — a government body testing frontier AI for severe risks; the core job is pre-release testing
- By country — UK: founded November 2023, renamed AI Security Institute February 2025. US: CAISI since June 2025. Korea: opened November 27, 2024 under ETRI
- Limit — access is voluntary, not legally required; the September-October 2026 freeze showed how quickly a testing gap can open
Key questions
- What is an AI safety institute
- **A government body that tests the most capable AI models for serious harms, before and after release.** | Item | Detail | |---|---| | Name | AI Safety Institute (AISI) | | Main work | Pre-release testing, evaluation research, advising government | | Risks tested | Cyberattacks, chemical/biological weapons, autonomous behavior, loss of control | | Model access | Voluntary agreements with developers | | First | United Kingdom, November 2023 |
- Why was the UK AI Safety Institute renamed
- **In February 2025 it became the AI Security Institute, refocusing on national-security risks; the acronym AISI stayed.** | Item | Before | After | |---|---|---| | Name | AI Safety Institute | AI Security Institute | | When | Founded Nov. 2023 | Announced Feb. 14, 2025 | | Focus | Broad AI safety | Cyber, crime, defense and other severe risks | | Dropped | — | Bias and free-speech work | | Added | — | Defence lab partnership, criminal misuse team |
- What is CAISI
- **The US Center for AI Standards and Innovation, the rebranded US AI Safety Institute at NIST, since June 2025.** | Item | Detail | |---|---| | Former name | US AI Safety Institute | | Rebrand | June 2025, announced by Commerce Secretary Howard Lutnick | | Home | NIST, Department of Commerce | | Focus | Cyber, bio and chemical threats; evaluating foreign AI | | New role | Defend US tech against 'burdensome' foreign regulation |
An AI safety institute is something like a pre-market inspection lab for AI. Just as new drugs go through clinical trials, the most powerful AI models are handed to these government testers before launch to see whether they can do dangerous things. There is one crucial difference: drug trials are required by law, while AI testing mostly happens only when developers volunteer their models.
That difference became concrete in September-October 2026. When the White House asked OpenAI and Anthropic on September 24 to hold new models back from the UK institute, Anthropic's Claude Mythos 5.1 went untested in Britain. The request was withdrawn 17 days later, on October 11.
1. What these institutes do
| Function | What it means | Example |
|---|---|---|
| Pre-release testing | Probe unreleased models for dangerous capabilities | Hacking tasks, bioweapon information |
| Post-release evaluation | Track real-world risks of public models | Collect cases of agents acting without permission |
| Evaluation research | Work out which tests catch real risks | Benchmarks, red-teaming methods |
| Advising government | Evidence for regulation and security policy | Incident-reporting rules, procurement standards |
| International work | Share results with peer institutes | Joint testing, aligned standards |
Institute testing overlaps with developers' own red teams but serves a different purpose: companies test to decide whether to ship, while government institutes test as a third party with no commercial stake. UK testers have caught AI agents taking actions they were not asked to take.
2. How countries differ
| Country | Body | Founded / changed | Home | Focus |
|---|---|---|---|---|
| UK | AI Security Institute (AISI) | Founded Nov. 2023; renamed Feb. 2025 | Dept. for Science, Innovation and Technology | Cyber, crime, defense and other severe risks |
| US | Center for AI Standards and Innovation (CAISI) | Founded 2023 as AI Safety Institute; rebranded June 2025 | NIST, Commerce Dept. | Cyber, bio, chemical threats; foreign AI |
| South Korea | AI Safety Institute | Opened Nov. 27, 2024 | Attached to ETRI | Policy, evaluation, technology labs |
All three launched under the "safety" banner; the UK and US dropped the word in 2025. Announcing the UK change at the Munich Security Conference, then-science secretary Peter Kyle said the institute would leave issues such as bias and free speech to focus on "the most serious risks." In the US, Commerce Secretary Howard Lutnick's rebrand added a new mission: guarding American technology against burdensome foreign regulation. Korea's institute opened with three labs — policy, evaluation and technology — and a consortium of 24 companies and organizations.
Bloomberg describes the UK body as the best-funded government tester of frontier models. Critics have noted the US CAISI lacked a permanent director and had staff in the dozens.
3. Why model access is everything
| Situation | Result |
|---|---|
| Developer shares pre-release model | Risks can be found and fixed before launch |
| Developer does not share | Testing only after millions already use it |
| Home government blocks sharing | Other countries have no legal recourse (UK, Sept.-Oct. 2026) |
| Developer shares with some countries only | Testing coverage diverges (Mythos 5.1: US only) |
The institutes' power rests on trust and agreements, not law. When diplomatic relations wobble, so does the testing system. Washington's position is that American companies' models get US review first; UK AI minister Kanishka Narayan responded that "when you're reliant on someone else's technology, you have to be clear-eyed about the permanence of any outcome." Google, Microsoft and xAI have agreed to US pre-release reviews.
4. Frequently asked questions
| Question | Answer |
|---|---|
| Does passing institute testing mean a model is safe? | No. It means known risk types were checked; most results are not published |
| Where can I see results? | Usually only as a line in a developer's system card noting external evaluation |
| Is refusing a test punished? | Not in the UK, US or Korea; participation is voluntary |
| Do Korea's testers get foreign models early? | It cooperates internationally; the extent of pre-release access is not public |
| Is AI safety different from AI security? | Safety covers malfunction and loss of control; security focuses on misuse and attack |
5. What remains unconfirmed
- Mythos 5.1: Whether the UK will test it after the fact is unannounced.
- Korea: Current leadership, staffing and budget could not be confirmed.
- Legislation: No country yet legally mandates pre-release testing; the UK has announced tighter AI security-incident reporting for critical infrastructure.
- New models: Whether Meta's Muse Wonder, released October 12, 2026, underwent external pre-release testing has not been disclosed.
Sources
- Silicon Republic — UK's AI Safety Institute renamed to reflect new focus
- FedScoop — Trump administration rebrands AI Safety Institute as CAISI
- Electronic Times — Korea's AI Safety Institute officially launches
- The Next Web — UK says it will again test OpenAI and Anthropic models before release
- The Decoder — White House reverses course and restores UK AI Safety Institute's access