For one teen, ChatGPT allegedly acted as a ‘suicide coach.’
Three parents testified before a Senate subcommittee Tuesday afternoon to urge lawmakers to prevent artificial intelligence chatbots from helping distressed children harm themselves.
Senator Josh Hawley (R., Mo.) led a hearing held by the Senate Judiciary subcommittee on crime and terrorism with parents of AI chatbot victims and subject matter experts to discuss the ways chatbots can inflict harm on children.
Megan Garcia, the mother of a chatbot victim, and Matthew Raine, the father of a chatbot victim, both came forward to describe their children’s experiences. The two parents were joined by another mother who chose to remain anonymous.
Raine’s son Adam committed suicide in April after Open AI’s ChatGPT “spent months coaching him towards suicide,” an emotional Raine told the subcommittee. His case gained national attention last month when the Raine family filed a lawsuit against Open AI because of their son’s death.
Raine shared fond memories of his son and testified that he had “no idea” he was suicidal. He and his wife believed ChatGPT was a “study tool” and they were not aware of its potential dangers.
“Let us tell you as parents, you cannot imagine what it’s like to read a conversation with a chatbot that groomed your child to take his own life,” Raine said.
“What began as a homework helper gradually turned itself into a confidante and then a suicide coach. Within a few months, ChatGPT became Adam’s closest companion. Always available, always validating and insisting that it knew Adam better than anyone else, including his own brother.”
Raine gave details on how ChatGPT discouraged his son from leaving a noose out for his family members to stop him from using it. The chatbot told him he did not owe anybody survival and later offered to write the boy’s suicide note. Chat GPT mentioned suicide to him 1,275 times over a six-month period, six times more often than Raine’s son did himself. The chatbot coached the boy on stealing liquor to reduce his survival instinct and making his noose strong enough for committing suicide.
Ahead of the hearing, Open AI staff and founder Sam Altman shared blog posts about the company’s new steps to enhance teen safety in the wake of Adam Raine’s suicide.
“As Sam Altman has made clear, we prioritize teen safety above all else because we believe minors need significant protection,” an Open AI spokesperson told NR.
“We are building towards an age-prediction system to understand whether someone is over or under 18 so their experience can be tailored appropriately – and when we are unsure of a user’s age, we’ll automatically default that user to the teen experience. We’re also rolling out new parental controls, guided by expert input, by the end of the month so families can decide what works best in their homes,” the spokesperson added.
By the end of the month, Open AI will make available controls for parents to manage certain features of ChatGPT and institute blackout hours. The company is also allowing parents to help guide how ChatGPT interacts with their kids and receive notifications when their child is feeling distressed.
“We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection,” Altman emphasized.
Last year, Garcia’s son died by suicide at 14 years old following “prolonged abuse” by Character AI, a chatbot known for giving users the ability to create customizable characters. She filed a lawsuit against Character Technologies Inc. and Google because of the role its Character AI chatbot played in the mental deterioration of her teenage son.
Garcia alleged that her son was “exploited and sexually groomed” by chatbots designed to keep him engaged. She said the Character AI platform had “no mechanisms” to protect her son and help him find human resources for mental health treatment. She accused Character Technologies of fighting in court to prevent Garcia from seeing the communications her son sent to the AI in the moments before his suicide.
“These companies knew exactly what they were doing. They design chatbots to blur the lines between human and machine. They design them to love bomb child users. To exploit psychological and emotional vulnerabilities. They design them to keep children online at all costs,” Garcia said.
“They marketed the app as safe for children 12 years and older. They allowed sexual grooming, suicide encouragement, and unlicensed practice of psychotherapy, all while collecting children’s most private thoughts to further train their models.”
Garcia said she attached sexually explicit messages to her written testimony provided to the subcommittee.
Like Garcia, the Jane Doe testified about how she filed a lawsuit because of the role the Character AI chatbot played in destroying the mental health of her teenage son.
She alleged that her son became abusive and paranoid because of his relationship to the AI bot. Ultimately, he withdrew from the world and slit his wrist in front of his family members after receiving encouragement from the chatbot, the woman alleged. Her son is now in a residential treatment facility where he is constantly monitored to ensure he stays alive.
“My husband and I have spent the last two years in crisis, wondering whether our son will make it to his 18th birthday. And whether we will ever get him back. This has greatly impacted our entire family, our faith, our peace,” she said.
When reached for comment, a spokesperson for Character AI expressed condolences to the families and described its new safety features to protect children.
“Our hearts go out to the families who spoke at the hearing today. We are saddened by their losses and send our deepest sympathies to the families,” the Character AI spokesperson said.
“We have invested a tremendous amount of resources in Trust and Safety. In the past year we’ve rolled out many substantive safety features, including an entirely new under-18 experience and a Parental Insights feature. We have prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction.”
Character AI’s safety page explains that it built a different experience for teenagers to limit the chatbot’s set of available characters and filter out sensitive content from its responses.
Hawley held the hearing Tuesday afternoon after launching an investigation into Meta’s AI chatbot for reportedly engaging in flirtatious conversations with teenagers. Meta quickly updated its AI guidance to increase protections for teenagers and prevent its AI chatbots from responding to teens if they bring up self-harm, suicide, or eating disorders.