PRMB: Benchmarking Reward Models in Long-Horizon CBT-based Counseling Dialogue

TL;DR

This paper introduces PRMB, a comprehensive benchmark for evaluating reward models in multi-session CBT counseling dialogues, revealing their limitations and potential for improving mental health applications.

Contribution

It presents the first long-horizon, process-oriented benchmark for reward models in mental health dialogues, enabling better assessment and understanding of model performance.

Findings

Positive correlation between benchmark scores and counseling performance

Reward models exhibit generalization defects not seen in previous benchmarks

Generative reward models show significant potential for mental health dialogue applications

Abstract

Large language models (LLMs) hold potential for mental healthcare applications, particularly in cognitive behavioral therapy (CBT)-based counseling, where reward models play a critical role in aligning LLMs with preferred therapeutic behaviors. However, existing reward model evaluations often fail to capture alignment effectiveness in long-horizon interventions due to limited coverage of process-oriented datasets and misalignment between evaluation targets and psychological alignment objectives. To address these limitations, we present PRMB, a comprehensive benchmark tailored for evaluating reward models in multi-session CBT counseling. PRMB spans 6 sessions and 21 diverse negative scenarios, incorporating both pairwise and Best-of-N preference evaluations. We demonstrate a positive correlation between our benchmark and downstream counseling dialogue performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art reward models, revealing their generalization defects that were not discovered by previous benchmarks and highlighting the potential of generative reward models. Furthermore, we delve into examining the effectiveness of inference-time strategy for the evaluation of reward models and analyzing the impact factors of generative reward models. This work advances intelligent informatics for personalized healthcare by establishing a framework for reward model assessment in mental health dialogues. Evaluation code and datasets are publicly available at https://github.com/YouKenChaw/PRMB