_COVER.png)
Why AI Detectors Don't Really Work (And What Does)
Quality reviewed September 2026
Facts verified September 20, 2026
- Subject
- Career & Life Skills
- Grades
- Grades 9-12
- Length
- 10-page student worksheet
- Time
- About one 50-minute class period
- Great for
- Sub plans and independent work
- Includes
- Reading passage, comprehension questions + full answer key
- Format
- Print-and-go PDF
What students do
- Amara's flagged essay traces how a detection tool misread her third-language writing as AI-generated
- Match perplexity and burstiness, the two metrics detectors measure, to what each one actually catches
- Simple edits defeat most detection tools while careless copy-pasting gets caught, and the reading explains the asymmetry
- Write short answers comparing process documentation and portfolio assessment to a single detector verdict
What it covers
- Perplexity and burstiness, the two statistics AI detectors actually measure
- Amara's case and the documented bias against non-native speakers and AAVE writers
- Process documentation and portfolio assessment as more reliable alternatives to a detector's verdict
Start reading this worksheet
The Detector Promise and the Reality
When AI writing tools like ChatGPT became widely available in late 2022, a predictable response emerged almost immediately: the development of AI detection tools. Companies including Turnitin, GPTZero, and others promised tools that could scan a piece of writing and return a verdict: human-written or AI-generated. For schools, employers, publishers, and anyone else concerned about the authenticity of text they were receiving, these tools seemed to offer a technological solution to a technological problem. The detection industry grew quickly, backed by real institutional demand. The reality of these tools has been significantly more complicated and, in many ways, more troubling than their marketing suggested. Extensive independent testing has consistently found that AI detection tools have substantial false positive rates: they flag genuinely human-written text as AI-generated at rates that make them unreliable as accusatory tools. Simultaneously, they can be defeated by relatively simple manipulation strategies that preserve the meaning of AI-generated text while changing its statistical patterns enough to evade detection. The gap between the promise and the performance of AI detection is large enough to matter in every context where these tools are used.
How AI Detectors Work, and Why They Fail
AI text detectors analyze statistical properties of text that differ between human writing and AI writing. The primary metric most detectors use is 'perplexity', a measure of how predictable or surprising the sequence of words in a text is. AI-generated text tends to be less perplexing than human text: AI models generate words by predicting the most statistically likely continuation of a sequence, which produces text that is statistically predictable in ways human writing typically is not. Lower perplexity is therefore a signal of potential AI generation. A related metric is 'burstiness', a measure of how variable the sentence lengths and complexity patterns in a text are. Human writing tends to be more variable: some sentences are short and punchy, others are long and complex, and this variability reflects genuine human compositional decision-making. AI writing tends toward more uniform sentence patterns. Detectors that measure both perplexity and burstiness perform better than those measuring only one metric. These metrics fail for several connected reasons. First, writing style varies enormously across individuals, contexts, and purposes. Formal academic writing by native English speakers already tends toward lower perplexity and more uniform sentence structures because that's what formal academic writing is trained to do. These writers are being penalized for writing well. Second, non-native English speakers often produce writing with lower lexical diversity and more uniform sentence patterns: not because they used AI, but because they are working in a second or third language where their options are more constrained. Third, as AI writing improves, its statistical patterns increasingly resemble human writing, narrowing the gap that detectors exploit.
The False Positive Problem in Detail
The false positive problem deserves more attention than it typically receives. A false positive in AI detection means that a real, human-written piece of text is flagged as likely AI-generated. The consequences in educational and professional settings can be severe: academic integrity violations, job loss, damaged reputations, and (in the case of students) failed assignments or courses. Multiple documented cases illustrate the real-world impact. In 2023, Turnitin's AI detection feature was found to have falsely flagged essays written by students with certain cognitive and learning differences whose writing style resembled AI output. Several universities reported cases where AI detection tools flagged the work of non-native English speakers at much higher rates than native speakers. A professor at Vanderbilt University documented testing the tool with his own published academic writing and receiving a flag suggesting portions were AI-generated. A study by researchers at the University of Pennsylvania found that AI detection tools disproportionately flag writing by African American students, whose use of African American Vernacular English (AAVE) apparently produces statistical patterns that trigger detection algorithms.
Like what you read? The full reading, the case study, and every question set are in the free download below, along with the answer key.
Why this matters
Plenty of students in this class are going to be second-language writers or simply careful, formal writers, exactly the profile the reading shows getting falsely flagged most often, so this is not a hypothetical fairness question for them. It also reframes the whole debate away from a technology fix and toward a better question, whether a student actually understands what they wrote, which is something a two-minute conversation about their draft can answer far more reliably than any detector score ever could.
From one teacher to another
Amara wrote her paper over three weeks in her third language and still got called into an academic integrity meeting after Turnitin flagged it as 58 percent likely AI-generated, and her case sits at the center of this reading. Students learn the two metrics detectors actually measure, perplexity and burstiness, and why formal writing by skilled writers, along with writing by non-native English speakers, already scores in the AI-like range on both. A section on evasion shows that simple edits defeat most detectors while careless copy-pasting gets caught, an asymmetry the reading calls genuinely unfair. It closes on what works better: drafts, notes, and portfolio assessment that a detector cannot replicate.
What you get
Download
Where should I send the next one?
Everything here is free and it stays that way. Leave your email and I'll send you the new lessons as they go up. You can unsubscribe from any one of them.
Related resources
AI in News and Misinformation Campaigns
AI-Written Text: How to Tell If a Human Wrote It
Building a Personal Verify Before You Share Habit
Deepfake Audio: The Voice-Cloning Threat
Lateral Reading: Verifying What You See Online
Reverse Image Search and Other Detection Tools
Every lesson on this site is free to download and use. If one of them saved you a prep period, you can chip in a few bucks toward my three kids' daycare fund. That's genuinely where tips go.