Preference Learning Unlocks LLMs' Psycho-Counseling Skills

TL;DR

This paper introduces a new preference dataset and reward model to enhance large language models' psycho-counseling abilities, achieving high performance aligned with professional standards.

Contribution

It creates PsychoCounsel-Preference, a large preference dataset, and develops PsychoCounsel-Llama3-8B, a model that significantly improves LLMs' psycho-counseling responses.

Findings

PsychoCounsel-Preference contains 36k high-quality preference pairs.

PsychoCounsel-Llama3-8B achieves an 87% win rate against GPT-4o.

The models and dataset are publicly released for further research.

Abstract

Applying large language models (LLMs) to assist in psycho-counseling is an emerging and meaningful approach, driven by the significant gap between patient needs and the availability of mental health support. However, current LLMs struggle to consistently provide effective responses to client speeches, largely due to the lack of supervision from high-quality real psycho-counseling data, whose content is typically inaccessible due to client privacy concerns. Furthermore, the quality of therapists' responses in available sessions can vary significantly based on their professional training and experience. Assessing the quality of therapists' responses remains an open challenge. In this work, we address these challenges by first proposing a set of professional and comprehensive principles to evaluate therapists' responses to client speeches. Using these principles, we create a preference dataset, PsychoCounsel-Preference, which contains 36k high-quality preference comparison pairs. This dataset aligns with the preferences of professional psychotherapists, providing a robust foundation for evaluating and improving LLMs in psycho-counseling. Experiments on reward modeling and preference learning demonstrate that PsychoCounsel-Preference is an excellent resource for LLMs to acquire essential skills for responding to clients in a counseling session. Our best-aligned model, PsychoCounsel-Llama3-8B, achieves an impressive win rate of 87% against GPT-4o. We release PsychoCounsel-Preference, PsychoCounsel-Llama3-8B and the reward model PsychoCounsel Llama3-8B-Reward to facilitate the research of psycho-counseling with LLMs at: https://hf.co/Psychotherapy-LLM.