PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Models

TL;DR

PERM introduces a psychology-grounded, bidirectional reward model for LLMs that improves empathetic responses by evaluating both supporter and seeker perspectives, outperforming existing methods in benchmarks and user preferences.

Contribution

This paper presents PERM, a novel bidirectional empathy evaluation framework grounded in psychology, enhancing LLMs' empathetic response generation beyond unidirectional models.

Findings

PERM outperforms state-of-the-art baselines by over 10% on benchmarks.

A blinded user study shows 70% preference for PERM-based responses.

PERM effectively incorporates supporter, seeker, and bystander perspectives.

Abstract

Large Language Models (LLMs) are increasingly deployed in human-centric applications, yet they often fail to provide substantive emotional support. While Reinforcement Learning (RL) has been utilized to enhance empathy of LLMs, existing reward models typically evaluate empathy from a single perspective, overlooking the inherently bidirectional interaction nature of empathy between the supporter and seeker as defined by Empathy Cycle theory. To address this limitation, we propose Psychology-grounded Empathetic Reward Modeling (PERM). PERM operationalizes empathy evaluation through a bidirectional decomposition: 1) Supporter perspective, assessing internal resonation and communicative expression; 2) Seeker perspective, evaluating emotional reception. Additionally, it incorporates a bystander perspective to monitor overall interaction quality. Extensive experiments on a widely-used emotional intelligence benchmark and an industrial daily conversation dataset demonstrate that PERM outperforms state-of-the-art baselines by over 10\%. Furthermore, a blinded user study reveals a 70\% preference for our approach, highlighting its efficacy in generating more empathetic responses. Our code, dataset, and models are available at https://github.com/ZhengWwwq/PERM.