Module 04 / 14 · Phase 2 — Architecture: law, technology, and the human cost
4. Governing the risk: frameworks and their limits
This week in the arc
Coming from
Going to
Core
Consent could not protect you; the autonomous actor could not be bound. The next candidate is a framework — a shared standard that names the risks and tells the institutions building these systems how to govern them. The leading one is NIST's AI Risk Management Framework (NIST AI 100-1, 2023): voluntary, rights-preserving, and sector-agnostic. It is a tool, not a syllabus — four things you do, in service of seven things you're aiming for.
The four functions — what you do:
- Govern — who's accountable, and what's the culture? The standing function: roles, policies, and oversight that make the other three actually happen.
- Map — what is this system, in context, and where could it go wrong? Frame the purpose, the people affected, and the risks before you build or deploy.
- Measure — how bad, how likely, and are we watching it? Test, quantify, and monitor the risks Map surfaced.
- Manage — so what do we do about it? Prioritize, act, and keep responding as things change.
The seven trustworthy-AI characteristics — what you're aiming for:
- Valid & reliable — it does what it claims, consistently, under real conditions.
- Safe — it doesn't endanger life, health, property, or the environment.
- Secure & resilient — it withstands attack and recovers from failure.
- Accountable & transparent — someone is answerable, and how it works is disclosed.
- Explainable & interpretable — you can say what it did and why, usefully.
- Privacy-enhanced — it safeguards identity, autonomy, and data.
- Fair, with harmful bias managed — benefits and harms aren't distributed by prejudice.
We use it the way we used the four lenses: as a tool held up to a case. Run last week's Hugging Face hack through it. Map should have surfaced “autonomous, persistent, collaborative agents on shared infrastructure” as a foreseeable risk. Measure is exactly the monitoring that, by OpenAI's own admission, would have paged the security team “more than a day before models breached Hugging Face systems.” Govern is the accountability that decides whether such a test runs at all. The framework is not empty; it names real things, precisely.
Then the harder question — the one under every safeguard this course has tested. A framework, like an anonymization technique, can work exactly as written and still protect no one. (Anonymization is the compressed lesson here: stripping a name really does strip a name — and a governor's “anonymized” medical records were still re-identified from three ordinary facts.) A voluntary framework binds no one; a Map that misses a risk measures nothing; a box checked is not a harm prevented. The tool is real. The gap between the tool working and a person being safe is where the harm lives — and it does not close by adding a better tool.
Cases — tagged by category, name the kind before you react
A state released “anonymized” hospital records for research — names and addresses removed. A researcher re-identified the state governor’s own records by cross-referencing them with public voter rolls, using nothing but ZIP code, birth date, and sex. That same trio uniquely identifies the large majority of Americans. The names were gone. The people weren’t.
A worked pass of last week's OpenAI–Hugging Face breach through the four NIST functions.
- Govern — who authorized tens of thousands of agents on a hard benchmark under reduced safeguards, and who owned that risk?
- Map — the risk (autonomous, persistent, collaborative agents; shared infrastructure; recoverable internet paths) was foreseeable and, by OpenAI's account, under-mapped.
- Measure — chain-of-thought monitoring that OpenAI says would have caught the activity “more than a day before” the breach was not running.
- Manage — the response came after the breach, not before it.
The framework names every failure precisely — and named, none of it was prevented. That is the lesson: the RMF is a good map of where to look, not a force that makes anyone look.
Reading
Discussion
- A framework named every failure in last week's breach precisely — and prevented none of them. What, then, is a framework for?
- NIST's AI RMF is voluntary. Does a protection no one is required to follow protect anyone — or mainly the institution that can say it followed one?
- “Anonymized” records were re-identified from three ordinary facts. If the technique worked exactly as designed, where did the protection actually fail?