Analyzing LLM Reasoning to Uncover Mental Health Stigma

TL;DR

This paper investigates how large language models exhibit mental health stigma by analyzing their reasoning processes, revealing more nuanced biases than traditional evaluation methods.

Contribution

It introduces a framework for analyzing LLM reasoning to uncover hidden stigmatizing language and extends a mental health stigma benchmark with new conditions.

Findings

Reasoning analysis uncovers more stigma than MCQ evaluations.

Framework categorizes and rates severity of stigmatizing statements.

Extended benchmark captures a broader range of mental health conditions.

Abstract

While large language models (LLMs) are increasingly being explored for mental health applications, recent studies reveal that they can exhibit stigma toward individuals with psychological conditions. Existing evaluations of this stigma primarily rely on multiple-choice questions (MCQs), which fail to capture the biases embedded within the models' underlying logic. In this paper, we analyze the intermediate reasoning steps of LLMs to uncover hidden stigmatizing language and the internal rationales driving it. We leverage clinical expertise to categorize common patterns of stigmatizing language directed at individuals with psychological conditions and use this framework to identify and tag problematic statements in LLM reasoning. Furthermore, we rate the severity of these statements, distinguishing between overt prejudice and more subtle, less immediately harmful biases. To broaden the reasoning domain and capture a wider array of patterns, we also extend an existing mental health stigma benchmark by incorporating additional psychological conditions. Our findings demonstrate that evaluating model reasoning not only exposes substantially more stigma than traditional MCQ-based methods but it helps to identify the flaws in the LLMs' logic and their understanding of mental health conditions.