PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators

TL;DR

This paper introduces PSI-Bench, an interpretable evaluation framework for depression patient simulators, revealing their limitations and guiding future improvements in mental health training tools.

Contribution

The paper presents PSI-Bench, a novel, clinically grounded evaluation framework that assesses depression simulators across multiple behavioral dimensions, addressing prior evaluation shortcomings.

Findings

Simulators tend to produce overly long, lexically diverse responses.

Simulators show reduced behavioral variability and quick emotion resolution.

The simulation framework impacts fidelity more than model size.

Abstract

Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulating depressed patients is particularly challenging, as safety constraints and high patient variability complicate simulations and underscore the need for simulators that capture diverse and realistic patient behaviors. However, existing evaluations heavily rely on LLM-judges with poorly specified prompts and do not assess behavioral diversity. We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically grounded diagnostics of depression patient simulator behavior across turn-, dialogue-, and population-level dimensions. Using PSI-Bench, we benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses, show reduced variability, resolve emotions too quickly, and follow a uniform negative-to-positive trajectory. We also show that the simulation framework has a larger impact on fidelity than the model scale. Results from a human study demonstrate that our benchmark is strongly aligned with expert judgments. Our work reveals key limitations of current depression patient simulators and provides an interpretable, extensible benchmark to guide future simulator design and evaluation.