Module 09 / 14 · Phase 4 — Automated judgment and the human on the other end
9. When the machine decides
This week in the arc
Coming from
Week 8 left a machine that can produce true facts about you no one can read. Now those facts — and everything else collected about you — start deciding things.
Going to
Week 10 — the same scoring turned on your wants and your future: profiling, prediction, and whether a system that models you can also move you.
Core
For eight weeks the machine watched, gathered, and — last week — produced. This week it does something new with all of it: it decides. Whether you get the interview, the loan, the apartment, the shorter sentence, the benefits that keep you housed — increasingly a model ranks you and a human signs off on the rank. Phase 4 is about being on the receiving end of automated judgment, and it opens where the stakes are oldest and highest: discrimination.
Start by killing the easy version. “Algorithms are biased” is true, and it is not the interesting claim — because bias is ancient. Humans have discriminated in hiring and lending and policing forever. If the story were only “the computer inherited our prejudice,” there would be nothing new to teach. Two things are new, and they are the week.
The first is laundering. A biased human can be argued with, appealed, held to account — and named as biased. Route the same decision through a model and the bias re-emerges wearing a disguise it has never had: the appearance of objectivity. “The algorithm scored you” sounds like a measurement, not a verdict. The discrimination didn’t vanish; it got credentialed.
The second is proxies. The law forbids deciding by race, or sex, or disability — so the model doesn’t use them. It uses zip code, shopping history, the name of your high school, the gap in your résumé, the phone in your pocket. Each is legal. Together they reconstruct the forbidden trait well enough to discriminate through it, and no line of code ever names the thing it is sorting you by. You cannot point to the prejudiced sentence, because there isn’t one — only a correlation, doing the work a slur used to do.
Now watch what that does to your protection. Anti-discrimination law is one of the strongest instruments in this whole course — but it was built to catch decisions made for reasons, by someone who could be asked why. A model that discriminates through proxies it found on its own, and cannot explain, may slip right between the law’s fingers. That gap — a real harm the best instrument we have might not reach — is the subject of the night. And underneath it, the question that makes the week genuinely hard: if the algorithm is measurably less biased than the humans it replaced, but no one can explain or appeal it, is that progress, or a better-dressed trap?
Cases — tagged by category, name the kind before you react
A risk score used across US health systems to flag patients for extra care rated Black patients as lower risk than white patients who were just as sick — steering care away from the people who needed it most, at the scale of millions. It never used race. It used past healthcare spending as a stand-in for health need — and because less money has historically been spent on Black patients’ care, the proxy quietly re-encoded the very disparity it should have corrected. No one designed it to discriminate. It discriminated anyway, and returned the result as a number.
A tool used in bail and sentencing rated defendants on their odds of reoffending. Investigative reporters found it wrong about Black defendants in the way that hurts — labelling them high-risk when they did not reoffend — far more often than white defendants. The company answered with its own analysis showing the tool was equally accurate across race. The unsettling part: both were right. They had used different, reasonable definitions of “fair,” and it is mathematically impossible to satisfy them all at once. The machine didn’t merely inherit a bias — it forced a choice about what fairness even means, and made that choice invisibly, on someone’s freedom.
Reading
Discussion
- A biased hiring manager can be named, argued with, and appealed. A model that scores you the same way mostly can't be any of those. Is the harm the same, worse, or just different — and why?
- The model never used race — it used zip code, spending, the phone you own. If the effect is identical to using race, does it matter, morally or legally, that it didn't?
- Two rigorous teams called the same risk tool biased and unbiased, using different definitions of 'fair.' If you had to build a system on exactly one definition, which — and who, specifically, do you make worse off by choosing it?
- Say the algorithm is measurably *less* biased than the humans it replaced, but no one can explain or appeal it. Is that a system you'd want deciding your loan? Your bail? Does your answer change depending on which one?