Measuring What Matters!! Assessing Therapeutic Principles in Mental-Health Conversation

TL;DR

This paper introduces a structured evaluation framework, CARE, for assessing AI-generated therapeutic responses against core psychotherapeutic principles, improving over baseline models and emphasizing clinical fidelity.

Contribution

The paper presents FAITH-M, a new benchmark with expert ratings, and CARE, a multi-stage reasoning framework that enhances evaluation of therapeutic principles in AI mental health responses.

Findings

CARE achieves an F-1 score of 63.34, outperforming the baseline Qwen3 by 64.26.

Structured reasoning and contextual modeling significantly improve evaluation accuracy.

The framework demonstrates robustness under domain shift and highlights challenges in modeling clinical nuances.

Abstract

The increasing use of large language models in mental health applications calls for principled evaluation frameworks that assess alignment with psychotherapeutic best practices beyond surface-level fluency. While recent systems exhibit conversational competence, they lack structured mechanisms to evaluate adherence to core therapeutic principles. In this paper, we study the problem of evaluating AI-generated therapist-like responses for clinically grounded appropriateness and effectiveness. We assess each therapists utterance along six therapeutic principles: non-judgmental acceptance, warmth, respect for autonomy, active listening, reflective understanding, and situational appropriateness using a fine-grained ordinal scale. We introduce FAITH-M, a benchmark annotated with expert-assigned ordinal ratings, and propose CARE, a multi-stage evaluation framework that integrates intra-dialogue context, contrastive exemplar retrieval, and knowledge-distilled chain-of-thought reasoning. Experiments show that CARE achieves an F-1 score of 63.34 versus the strong baseline Qwen3 F-1 score of 38.56 which is a 64.26 improvement, which also serves as its backbone, indicating that gains arise from structured reasoning and contextual modeling rather than backbone capacity alone. Expert assessment and external dataset evaluations further demonstrate robustness under domain shift, while highlighting challenges in modelling implicit clinical nuance. Overall, CARE provides a clinically grounded framework for evaluating therapeutic fidelity in AI mental health systems.