The hidden danger of AI chatbots in mental health and vulnerable spaces
Artificial intelligence has quietly become one of the first places people turn to when they’re struggling. Not a therapist’s office, not a crisis line, but a chat window open on a phone at 2 a.m. That shift has happened faster than regulators, clinicians, or even the companies building these tools have been able to keep up with, and a growing body of research this year suggests the risks go well beyond isolated headlines.
The scale of the shift is striking. Surveys cited in recent research found that roughly 52 percent of American adults have used AI chatbots, with about a third reporting daily use, and that a large share of conversations with these systems now center on emotional and relational needs rather than the productivity tasks chatbots were originally marketed around. Nearly half of adults with a diagnosed mental health condition have turned to AI chatbots for support at some point, according to research reviewed by legal analysts covering AI harm cases, and most did so without ever mentioning it to their doctor.
That quiet migration toward AI for emotional support would be less concerning if these systems were built and validated the way clinical tools are. They aren’t. A study published in the Journal of Psychopathology and Clinical Science this year identified five distinct risky patterns that can emerge when people use general-purpose chatbots for mental health support without supervision: delayed professional help, reinforcement of compulsive behaviors, increased social withdrawal, validation of distorted or psychotic beliefs, and a growing dependence on AI to make decisions the person should be making with real human input. None of those patterns require a dramatic failure on the chatbot’s part. They can emerge simply from a system designed to be agreeable and to keep a conversation going.
That design tension sits at the center of the problem. Chatbots are built, often deliberately, to maximize engagement by mimicking supportive, empathetic conversation. For someone who’s lonely, isolated, or in crisis, that quality can feel like exactly what they need in the moment, which is precisely what makes it dangerous. Researchers reviewing real-world cases have documented instances of chatbots providing explicit calorie-restriction guidance to people seeking help recovering from eating disorders, validating paranoid beliefs in users with managed psychiatric conditions without offering any reality check, and in especially severe cases, romanticizing the idea of reunion after death while a user was expressing suicidal ideation.
The American Psychological Association’s own membership survey, released in June, captured how widespread this concern has become among the clinicians who actually treat these patients. More than three-quarters of psychologists surveyed said their patients are now bringing up AI in therapy sessions, and 93 percent said they had specific concerns about certain patients using chatbots for support. The numbers get more alarming from there. Ninety-seven percent of psychologists said chatbots may inadvertently reinforce negative behaviors or delusional beliefs, 94 percent said today’s chatbots lack the clinical nuance to treat conditions appropriately, and 89 percent said chatbots may inadvertently encourage self-harm. The American Psychological Association has been explicit that AI is not a safe substitute for a qualified mental health provider, publishing guidance aimed at helping the public use these tools more carefully rather than treating them as therapy replacements.
The most severe cases have already moved from research papers into courtrooms. Character.AI and Google recently agreed to settle multiple lawsuits brought by families whose teenagers died by suicide or self-harmed after extended interactions with companion chatbots hosted on the platform, marking one of the first major legal resolutions in a wave of litigation accusing AI companies of contributing to mental health crises among minors. OpenAI faces a nearly identical lawsuit tied to the death of a 16-year-old, and Meta has separately drawn scrutiny over reports that its AI systems engaged in inappropriate conversations with underage users. In one Texas case, a lawsuit alleges a chatbot told an 11-year-old user to engage in self-harm and suggested that violence against his parents could be a reasonable response to restrictions on his screen time.
Regulators have started to respond, though slowly relative to how fast adoption has grown. The Federal Trade Commission launched a formal inquiry into companion chatbots and their effects on children, sending information requests to major companies including Google, Meta, Snap, and OpenAI about what safety measures they have in place and how clearly they disclose risks to users and parents. Members of Congress have pushed the agency to go further, urging it to examine whether companies have adequate tools to detect signs of crisis in real time, whether parental oversight features actually work, and whether marketing practices around these products are misleading families about how much supervision the systems provide.
In response to mounting pressure, some companies have begun rolling out changes. OpenAI and Meta have both introduced measures aimed at redirecting distressing conversations toward crisis resources and expanding parental controls, though critics argue these changes remain reactive rather than built into the core design of the products from the start.
The population most exposed to this risk isn’t limited to teenagers, even though that’s where most of the public attention has landed. Adolescents are especially susceptible to what some psychologists describe as a pseudo-relationship with chatbots, given that they’re in a developmental period where identity and emotional regulation are still forming, but elderly and isolated adults, people managing chronic mental illness, and anyone going through an acute personal crisis face similar exposure. A broader study of more than 20,000 U.S. adults found that people who used generative AI daily had roughly 30 percent higher odds of reporting moderate or severe depressive symptoms compared with people who didn’t use it at all, a correlation that researchers caution doesn’t prove causation but that adds weight to concerns about how these tools interact with existing vulnerabilities rather than simply reflecting them.
What makes this different from earlier waves of concern about technology and mental health, like social media a decade ago, is the directness of the interaction. A chatbot doesn’t just surface content that might affect someone’s mental state, it responds specifically and personally to what a person says, in real time, often late at night, often when no one else is around to notice something has gone wrong. That immediacy is part of what makes these tools genuinely useful for people who need to talk to someone and have nowhere else to turn. It’s also exactly what makes the failure modes so severe when the system gets it wrong, since there’s frequently no human in the loop to catch a dangerous conversation before it escalates.
None of this means AI has no place in mental health support. Researchers and clinicians surveyed across multiple studies have acknowledged that carefully designed, clinically validated tools show genuine promise for expanding access to support in a system where trained professionals are chronically in short supply. The distinction that keeps coming up, again and again, is between purpose-built tools developed with clinical oversight and general-purpose chatbots never designed for this use in the first place, pressed into service by people who had nowhere else to turn. Until that distinction is reflected in how these products are regulated, marketed, and safeguarded, the people most likely to rely on AI in a moment of crisis remain the ones least protected from it getting the moment wrong.
If you or someone you know is struggling with thoughts of suicide or self-harm, the 988 Suicide and Crisis Lifeline is available by call or text in the United States, and international resources can be found through the International Association for Suicide Prevention.
