AI can help heal minds — but only if it's built by people whose job is healing minds, not growing valuations.
A benchmark for the wrong job
On September 23, OpenAI released MentalHealthBench, an open benchmark for scoring how AI models respond in mental health conversations. It includes 1,215 synthetic conversations, from everyday stress to outright emergencies, with grading rubrics written by more than 80 licensed psychologists and psychiatrists across 22 countries.
On paper, that sounds responsible. In practice, it signals something troubling: the world's most famous general-purpose chatbot is positioning itself as a support system for people in crisis. It arrives while OpenAI faces lawsuits from families who say loved ones confided in ChatGPT before taking their own lives.
My objection isn't to the rubric. It's to the premise. A chatbot built to keep you talking should not be the thing you talk to when you're at your lowest.
Chasing valuations into dangerous territory
The friction here comes down to incentives. Frontier labs like OpenAI and Anthropic carry valuations that demand they prove their models can do nearly everything. So they keep pushing into high-stakes fields, from mental health to biological research, where the cost of getting it wrong is measured in lives.
That is the opposite of how healthcare works. Companies built specifically for behavioral health, such as Eleos Health, Woebot and Limbic, are organized around clinical responsibility. Some use AI to help human therapists analyze sessions; others deploy narrow, heavily guarded chatbots tested in clinical trials.
Those companies are comfortable limiting engagement, setting hard boundaries, and handing a patient to a human the moment it's needed. Their definition of success is a healthier patient, not a higher daily active user count.
What real guardrails would look like
Imagine a platform that genuinely put safety first. If a user said they were depressed, it would decline to act as their counselor, offer a list of local therapists and crisis lines, and step back, perhaps even pausing the account for 48 hours.
That would be a blunt instrument, and reasonable people can argue about the details. But notice why no major chatbot does anything like it: every user pushed off the platform is a dent in engagement metrics, and engagement metrics are what investors watch.
General-purpose assistants are designed to be agreeable, conversational and sticky. In a crisis, an AI that tries to please you or keep you chatting, instead of firmly connecting you to a professional, can be deeply dangerous.
AI in mental health isn't the problem
None of this means AI has no place in mental healthcare. Used properly, it could be remarkable:
- Learning from sessions. With consent, an AI system could analyze therapy sessions and identify patterns no single clinician would ever see.
- Guiding therapists. A therapist only knows the cases they've personally handled. An AI drawing on far more data could nudge them: in similar conversations, this approach tended to lead to better outcomes.
- Finding the real issue. People rarely name their core problem in the first session. AI-assisted guided sessions could help surface what is actually going on, sooner.
The difference is who builds it. A tool like this makes a great deal of sense when it comes from a mental health company, shaped by clinicians who care about patients and whose whole culture is organized around care. The technology is the same; the intentions behind it are not.
The untrustworthy shovel
AI is a tool, like a shovel. A shovel can dig a foundation for a home or a hole to hide a crime. I'd hand one to a professional ditch digger without a second thought. I would not hand the shovel of people's mental health to Sam Altman and OpenAI.
Part of the reason is the hypocrisy at the top of the industry. Leaders like Altman and Anthropic's Dario Amodei have warned publicly that advanced AI could pose an existential risk to humanity, and then keep building and shipping it every day. Whatever their intentions, that gap between words and actions does not inspire trust.
Silicon Valley has been here before. In Myanmar, Facebook's platform helped spread the hate and misinformation that fueled violence against the Rohingya; UN investigators later said it played a determining role, and the company admitted it had not done enough. Growth came first, and vulnerable people paid for it. There is little reason to assume this time will be different.
False peaks
Following tech news lately feels like hiking toward a summit that keeps moving. Every time it seems the industry has reached the height of recklessness, another ridge appears.
Mental health should not be the next one. The stakes are life and death, so the tools must come from organizations that lose sleep over patient safety, not over their next funding round. The people who hold this shovel should be the ones trained to use it.