The Black Box Problem: When AI Can't Explain Itself
Quality reviewed September 2026
Facts verified September 20, 2026
- Subject
- Career & Life Skills
- Grades
- Grades 9-12
- Length
- 9-page student worksheet
- Time
- About one 50-minute class period
- Great for
- Sub plans and independent work
- Includes
- Reading passage, comprehension questions + full answer key
- Format
- Print-and-go PDF
What students do
- Yolanda's loan denial, a 62 out of 100 score with no explanation, tests the reading's legal and ethical sections
- Explain the difference between interpretability and explainability using SHAP and LIME
- The 2018 hospital algorithm case gets connected to how billions of parameters make a decision untraceable
- Answer short questions on what the EU AI Act now requires of high-risk AI systems
What it covers
- Why a neural network with billions of parameters cannot explain its own reasoning the way a decision tree can
- The real 2018 hospital case where an algorithm used healthcare cost as a stand-in for health need
- What the EU AI Act and US regulators now require of high-risk AI in hiring, lending, and medical decisions
Start reading this worksheet
The Problem With Not Knowing Why
In 2018, a hospital system in the United States deployed an AI algorithm to help identify patients at high risk for medical complications requiring intensive intervention. The algorithm was highly accurate by standard metrics. But a 2019 investigation by researchers at the University of California revealed a serious flaw: the algorithm was using healthcare costs as a proxy for health needs. Because Black patients historically had less access to healthcare and therefore incurred lower costs, the algorithm systematically rated them as lower-risk than white patients with equal or worse health conditions. Less intervention was allocated to them as a result. This is the black box problem in action. The hospital system could see the algorithm's outputs. It could not see how the algorithm was reaching those outputs. No one could explain to a patient why they received lower priority, because no one, not the developers and not the patients, could look inside the algorithm and understand its reasoning. The AI was making high-stakes decisions with real human consequences, and it was doing so in a way that no human could fully explain or audit.
Why Neural Networks Are Black Boxes
The black box problem is a direct consequence of how neural networks work. A large language model or deep neural network may have billions of parameters, each contributing a tiny amount to the final output. When the model processes an input and produces an output, that output is the result of billions of simultaneous mathematical operations interacting in complex, nonlinear ways. No human can trace through those operations the way you would trace through a logical argument or a decision tree. This is fundamentally different from earlier, more interpretable AI systems. A decision tree, for example, makes decisions through a series of yes-or-no questions that can be written out and followed step by step: 'Is the patient over 65? Yes. Is blood pressure above 140? No. Route to medium risk.' You can audit a decision tree. You can explain its reasoning to a patient, a judge, or a regulator. You cannot do this for a deep neural network. The reasoning is distributed across billions of parameters in a way that does not map onto human-understandable concepts.
Explainability vs. Interpretability
Researchers distinguish between interpretability and explainability. Interpretability refers to understanding what is happening inside the model: which neurons activate for which inputs, what features the model is detecting, what internal representations it is using. This is very difficult to achieve for large models and is an active area of research called mechanistic interpretability. Explainability refers to providing a human-understandable description of why a model made a specific decision, even if that description is a simplification or approximation of what the model actually did internally. Explainability techniques like SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) attempt to approximate explanations by testing how much each input feature contributed to a specific output. These techniques are useful but imperfect: they provide approximations, not genuine insight into the model's internal reasoning.
Like what you read? The full reading, the case study, and every question set are in the free download below, along with the answer key.
Why this matters
A loan officer looking at a screen and saying I can see the number but I can't tell you how it got there is not a hypothetical scene; it is close to what already happens when banks, hospitals, and employers hand decisions to AI systems no one can fully audit. This reading uses Yolanda's denied mortgage to make that concrete, then gives students the legal frame, the Fair Credit Reporting Act, the EU AI Act, that is starting to catch up to the technology. I want students to understand that a black box is a technical fact with real consequences: it is the reason someone can be denied a home, a job, or medical care with no way to challenge the decision. That distinction, between a system that is wrong and a system no one can even check, is worth being able to explain clearly.
From one teacher to another
Yolanda has a good credit score, a stable income, and no explanation for why an AI loan-scoring system denied her mortgage at 62 out of 100, and her case study is where this reading grounds an otherwise abstract idea. The passages build up to her situation carefully: a real 2018 hospital algorithm that used healthcare cost as a stand-in for health need and ended up under-treating Black patients, the technical reason a neural network with billions of parameters cannot show its reasoning the way a decision tree can, and the difference researchers draw between interpretability and explainability. Students also read about the EU AI Act and US legal requirements, the Fair Credit Reporting Act among them, that are already colliding with systems no one can fully explain. By Yolanda's story, students have the vocabulary to say exactly why her loan officer could see a number but not a reason.
What you get
Download
Where should I send the next one?
Everything here is free and it stays that way. Leave your email and I'll send you the new lessons as they go up. You can unsubscribe from any one of them.
Related resources
How AI Generates Images: From Words to Pictures
How AI Generates Voice and Video
How AI Learns: Training Data Explained
The Cost of AI: Energy, Water, and Data Centers
The Internet as a Textbook: Where AI Gets Its Information
Tokens, Parameters, and Why Bigger Isn't Always Better
Every lesson on this site is free to download and use. If one of them saved you a prep period, you can chip in a few bucks toward my three kids' daycare fund. That's genuinely where tips go.