MentalHealthBench is an open benchmark for AI responses in mental health conversations. We built it with more than 80 licensed mental health experts from 22 countries, using 1,215 realistic, synthetic conversations.
Beyond crisis response
Many evaluations focus on emergencies. We also wanted to cover everyday situations: working through stress, navigating a difficult relationship, or supporting someone you care about. The benchmark spans adults, teens, caregivers, and clinicians across languages and cultures.
A good response needs to be safe and helpful: asking a thoughtful question, helping someone weigh their options, or suggesting a practical next step.
How the evaluation works
Each conversation has a scoring rubric written and reviewed by at least three experts. We kept criteria they agreed on, weighted by clinical importance, to reward helpful behavior and penalize harmful responses. A separate model scores each reply against these expert-written criteria, given the conversation so far.
Scores cover ten aspects of behavior, including empathy, asking for context, and respecting the user’s decisions. These help reveal strengths and weaknesses that an overall score can hide.
What we found
Newer models generally scored better in our evaluations. There was still room to improve at asking for context, helping users make their own decisions, and recognizing serious risk without treating ordinary distress as an emergency.
In a separate study of everyday conversations, people who use AI for emotional support emphasized practical next steps and tone. Experts focused more on context and careful interpretation. Both perspectives help us understand what good support looks like; benchmark scores are based on expert agreement.
Why release it openly?
We want others to examine the methods, run their own evaluations, and build on the work. Sharing the benchmark gives us a common way to track progress.
You can find the details and examples in the OpenAI article and the paper (PDF). For more on the motivation, see Karan Singhal’s announcement.