Trust Modeling in Counseling Conversations: A Benchmark Study

TL;DR

This paper introduces MENTAL-TRUST, a new dataset and benchmark for modeling trust in counseling conversations, aiming to quantify and analyze the therapeutic bond through textual interaction analysis.

Contribution

It presents the first annotated dataset with trust levels, formulates trust modeling as an ordinal classification task, and evaluates multiple language models on this new benchmark.

Findings

Trust levels can be effectively modeled as an ordinal classification task.

State-of-the-art language models show varying performance in trust prediction.

The dataset enables analysis of trust evolution in therapeutic interactions.

Abstract

In mental health counseling, a variety of earlier studies have focused on dialogue modeling. However, most of these studies give limited to no emphasis on the quality of interaction between a patient and a therapist. The therapeutic bond between a patient and a therapist directly correlates with effective mental health counseling. It involves developing the patient's trust on the therapist over the course of counseling. To assess the therapeutic bond in counseling, we introduce trust as a therapist-assistive metric. Our definition of trust involves patients' willingness and openness to express themselves and, consequently, receive better care. We conceptualize it as a dynamic trajectory observable through textual interactions during the counseling. To facilitate trust modeling, we present MENTAL-TRUST, a novel counseling dataset comprising manual annotation of 212 counseling sessions with first-of-its-kind seven expert-verified ordinal trust levels. We project our problem statement as an ordinal classification task for trust quantification and propose a new benchmark, TrustBench, comprising a suite of classical and state-of-the-art language models on MENTAL-TRUST. We evaluate the performance across a suite of metrics and lay out an exhaustive set of findings. Our study aims to unfold how trust evolves in therapeutic interactions.