From Measurement to Expertise: Empathetic Expert Adapters for Context-Based Empathy in Conversational AI Agents

TL;DR

This paper presents a novel framework for developing context-specific empathetic large language models, utilizing expert adapters to improve empathy alignment in conversational AI, significantly reducing empathy gaps and enhancing consistency over multi-turn interactions.

Contribution

It introduces empathetic expert adapters trained for varying empathy levels, along with a synthetic conversational pipeline, to improve context-based empathy in LLMs, a novel approach in this domain.

Findings

Achieved a 72.66% reduction in perceived empathy gap.

Scores increased by an average factor of 2.43.

Outperformed system prompts in maintaining empathy over turns.

Abstract

Empathy is a critical factor in fostering positive user experiences in conversational AI. While models can display empathy, it is often generic rather than tailored to specific tasks and contexts. In this work, we introduce a novel framework for developing and evaluating context-specific empathetic large language models (LLMs). We first analyze a real-world conversational dataset consisting of 672 multi-turn conversations across 8 tasks, revealing significant differences in terms of expected and experienced empathy before and after the conversations, respectively. To help minimize this gap, we develop a synthetic multi-turn conversational generation pipeline and steer responses toward our defined empathy patterns based on the context that more closely matches users' expectations. We then train empathetic expert adapters for context-specific empathy that specialize in varying empathy levels based on the recognized task. Our empirical results demonstrate a significant gap reduction of 72.66% between perceived and desired empathy with scores increasing by an average factor of 2.43 as measured by our metrics and reward models. Additionally, our trained empathetic expert adapters demonstrate superior effectiveness in preserving empathy patterns throughout conversation turns, outperforming system prompts, which tend to dramatically diminish in impact as conversations lengthen.