PsychEval: A Multi-Session and Multi-Therapy Benchmark for High-Realism AI Psychological Counselor

TL;DR

PsychEval introduces a comprehensive, multi-session, multi-therapy benchmark dataset designed to train and evaluate highly realistic AI psychological counselors with longitudinal memory, adaptive reasoning, and diverse therapeutic strategies.

Contribution

We present PsychEval, a novel benchmark with multi-session, multi-therapy data, extensive skill annotations, and a holistic evaluation framework for developing and assessing realistic AI psychological counselors.

Findings

Dataset includes over 677 meta-skills and 4577 atomic skills.

Supports five distinct therapeutic modalities and an integrative therapy framework.

Establishes a comprehensive evaluation with 18 metrics across client and counselor levels.

Abstract

To develop a reliable AI for psychological assessment, we introduce \texttt{PsychEval}, a multi-session, multi-therapy, and highly realistic benchmark designed to address three key challenges: \textbf{1) Can we train a highly realistic AI counselor?} Realistic counseling is a longitudinal task requiring sustained memory and dynamic goal tracking. We propose a multi-session benchmark (spanning 6-10 sessions across three distinct stages) that demands critical capabilities such as memory continuity, adaptive reasoning, and longitudinal planning. The dataset is annotated with extensive professional skills, comprising over 677 meta-skills and 4577 atomic skills. \textbf{2) How to train a multi-therapy AI counselor?} While existing models often focus on a single therapy, complex cases frequently require flexible strategies among various therapies. We construct a diverse dataset covering five therapeutic modalities (Psychodynamic, Behaviorism, CBT, Humanistic Existentialist, and Postmodernist) alongside an integrative therapy with a unified three-stage clinical framework across six core psychological topics. \textbf{3) How to systematically evaluate an AI counselor?} We establish a holistic evaluation framework with 18 therapy-specific and therapy-shared metrics across Client-Level and Counselor-Level dimensions. To support this, we also construct over 2,000 diverse client profiles. Extensive experimental analysis fully validates the superior quality and clinical fidelity of our dataset. Crucially, \texttt{PsychEval} transcends static benchmarking to serve as a high-fidelity reinforcement learning environment that enables the self-evolutionary training of clinically responsible and adaptive AI counselors.