{\Psi}-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback

TL;DR

{ extPsi}-Arena is an interactive framework for assessing and improving LLM-based psychological counselors through realistic simulations, multi-perspective evaluations, and iterative feedback-driven optimization, enhancing their efficacy and safety in mental health support.

Contribution

It introduces a comprehensive, interactive assessment framework with tripartite evaluation and closed-loop optimization for LLM-based mental health counselors, addressing limitations of prior static and uni-perspective methods.

Findings

Significant performance variation among state-of-the-art LLMs in realistic counseling scenarios.

Reflection-based optimization achieves up to 141% improvement in counseling effectiveness.

Multi-perspective evaluation reveals diverse strengths and weaknesses of LLM counselors.

Abstract

Large language models (LLMs) have shown promise in providing scalable mental health support, while evaluating their counseling capability remains crucial to ensure both efficacy and safety. Existing evaluations are limited by the static assessment that focuses on knowledge tests, the single perspective that centers on user experience, and the open-loop framework that lacks actionable feedback. To address these issues, we propose {\Psi}-Arena, an interactive framework for comprehensive assessment and optimization of LLM-based counselors, featuring three key characteristics: (1) Realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients, (2) Tripartite evaluation that integrates assessments from the client, counselor, and supervisor perspectives, and (3) Closed-loop optimization that iteratively improves LLM counselors using diagnostic feedback. Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives. Moreover, reflection-based optimization results in up to a 141% improvement in counseling performance. We hope PsychoArena provides a foundational resource for advancing reliable and human-aligned LLM applications in mental healthcare.