PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation

TL;DR

PsycoLLM is a specialized large language model trained on a high-quality psychological dataset, designed to improve understanding and evaluation in mental health applications, and evaluated with a comprehensive Chinese psychological benchmark.

Contribution

The paper introduces PsycoLLM, a novel psychological LLM trained on a curated dataset and evaluated with a new comprehensive psychological benchmark.

Findings

PsycoLLM outperforms other LLMs on the psychological benchmark.

The dataset and pipeline improve relevance and quality of psychological dialogue data.

PsycoLLM demonstrates superior performance in professional ethics and case analysis.

Abstract

Mental health has attracted substantial attention in recent years and LLM can be an effective technology for alleviating this problem owing to its capability in text understanding and dialogue. However, existing research in this domain often suffers from limitations, such as training on datasets lacking crucial prior knowledge and evidence, and the absence of comprehensive evaluation methods. In this paper, we propose a specialized psychological large language model (LLM), named PsycoLLM, trained on a proposed high-quality psychological dataset, including single-turn QA, multi-turn dialogues and knowledge-based QA. Specifically, we construct multi-turn dialogues through a three-step pipeline comprising multi-turn QA generation, evidence judgment, and dialogue refinement. We augment this process with real-world psychological case backgrounds extracted from online platforms, enhancing the relevance and applicability of the generated data. Additionally, to compare the performance of PsycoLLM with other LLMs, we develop a comprehensive psychological benchmark based on authoritative psychological counseling examinations in China, which includes assessments of professional ethics, theoretical proficiency, and case analysis. The experimental results on the benchmark illustrate the effectiveness of PsycoLLM, which demonstrates superior performance compared to other LLMs.