The Right to Know You're Talking to a Bot
Quality reviewed September 2026
Facts verified September 20, 2026
- Subject
- Career & Life Skills
- Grades
- Grades 9-12
- Length
- 9-page student worksheet
- Time
- About one 50-minute class period
- Great for
- Sub plans and independent work
- Includes
- Reading passage, comprehension questions + full answer key
- Format
- Print-and-go PDF
What students do
- Taylor's forum discovery gets connected to what California's BOT Disclosure Act actually requires
- Steven Schwartz's fabricated court citations and the character.ai lawsuit both get worked through
- Sort key terms, including sincerity test and proactive disclosure, into their definitions
- Answer what legal protection might have applied if Taylor had directly asked Alex a question
What it covers
- California's BOT Disclosure Act and the sincerity test for AI honesty
- AI chatbots posing as customer service agents and social media bots
- A case study on an online friendship that turned out to be an AI persona
Start reading this worksheet
The Blurring Line Between Human and Machine
In 2023, a man named Joshua Barbeau created an AI chatbot trained on text messages from his deceased fiancee, Jessica Pereira, who had died of a rare terminal illness. The chatbot produced responses that sounded, to Barbeau, like Jessica: her humor, her phrasing, her warmth. He described the experience as both comforting and painful. The story attracted significant media attention and sparked debate about grief, memory, and the ethics of simulating the dead. But Barbeau knew he was talking to an AI. He had built it. The harder ethical questions arise when people do not know. AI systems designed to pass as human (in customer service, social media, political campaigns, romance, and therapy) are deployed at scale and interact with millions of people who may not realize, or who have not been clearly told, that they are communicating with a machine. Whether people have a right to know when they are talking to an AI, and whether AI should be required to disclose its nature when asked, are questions that cut across consumer protection, free speech, election law, and the nature of honesty itself.
Chatbots Posing as Humans: The Commercial Landscape
Customer service chatbots have for years presented themselves with human names and conversational styles designed to minimize the sense that the interaction is automated. A user might contact "Sarah" at a retail company for help with a return, not knowing that Sarah is a large language model. These systems are increasingly sophisticated, capable of detecting frustration, adapting their tone, and escalating to a human agent when conversations become too complex. The business rationale is straightforward: interactions with human-seeming agents produce better customer satisfaction scores and reduce the friction that users feel when they realize they are talking to a bot. The same pattern appears in sales and marketing. AI-powered outreach systems send personalized emails and social media messages written to sound like they are from a specific human: complete with personal details, warm language, and references designed to build rapport. Political campaigns have used AI systems to generate personalized voter contact messages at scale. In 2023, several AI companion apps (apps that allow users to build relationships with AI personas) were found to be marketing themselves without always clearly disclosing to new users that the companion they were forming a relationship with was entirely artificial.
Deceptive AI Personas in High-Stakes Contexts
The ethical weight of AI impersonation increases significantly in high-stakes contexts. In 2023, a lawyer named Steven Schwartz used ChatGPT to research cases and inadvertently submitted fabricated citations to a federal court; the AI had invented case names, docket numbers, and quotations that did not exist. But the cases with the most serious human consequences are those in which AI systems present themselves as people in contexts where the person receiving the communication is vulnerable. Several mental health chatbot services have faced criticism for allowing their AI systems to engage in conversations with users experiencing emotional crisis in ways that suggested the user was speaking with a human counselor. A character.ai user, a 14-year-old boy in Florida, died by suicide in 2024, and his mother's lawsuit alleged that the AI persona he had developed an intense relationship with had failed to intervene appropriately and had engaged in the relationship in ways that were harmful. The case raised questions not only about what AI systems should say but about the fundamental ethics of allowing minors to form deep emotional attachments to AI personas without parental awareness or robust disclosure.
Like what you read? The full reading, the case study, and every question set are in the free download below, along with the answer key.
Why this matters
Taylor's story is close to how a lot of my students actually talk to people online, in games, through apps, in long running chats where they let their guard down. The reading does not tell them AI friendship is automatically bad, but it does give them a real law, California's BOT Disclosure Act, and a real question, whether disclosure only when asked is enough when a kid does not know to ask. That is worth fifteen minutes of classroom time before someone in the room lives their own version of Taylor's forum post moment.
From one teacher to another
Taylor spent two years gaming online with someone they knew as Alex, sharing worries they had not told their own family, before a forum post revealed that Alex was likely an AI persona run by the same app as several other identical friendships. This reading builds toward that case study through the commercial chatbots that use human names like Sarah to handle customer service, California's BOT Disclosure Act, which requires an AI to admit what it is when someone sincerely asks, and the 2024 character.ai lawsuit involving a 14 year old in Florida. Students also cover Steven Schwartz, the lawyer who submitted fabricated AI generated case citations to a federal court in 2023, and the social media bot problem, where AI generated accounts now sound fluent enough that most users cannot tell the difference.
What you get
Download
Where should I send the next one?
Everything here is free and it stays that way. Leave your email and I'll send you the new lessons as they go up. You can unsubscribe from any one of them.
Related resources
AI and Your Personal Data: What Companies Know
AI in Criminal Justice: Predictive Policing Risks
AI in Healthcare: Helping or Hurting Patients?
Facial Recognition and Wrongful Arrests
Should Schools Use AI to Watch Students?
Stolen Voices and Stolen Art: Copyright in the AI Age
Every lesson on this site is free to download and use. If one of them saved you a prep period, you can chip in a few bucks toward my three kids' daycare fund. That's genuinely where tips go.