EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World

TL;DR

EmoHarbor introduces an automated framework for evaluating personalized emotional support by simulating users' internal worlds, revealing that current models lack true user-awareness despite generating empathetic responses.

Contribution

The paper presents EmoHarbor, a novel evaluation framework that models user internal states and assesses personalized support, shifting focus from generic empathy to user-specific emotional support.

Findings

LLMs excel at empathetic responses but lack personalization.

Current models do not adequately consider individual user contexts.

EmoHarbor enables scalable, reproducible evaluation of personalized support.

Abstract

Current evaluation paradigms for emotional support conversations tend to reward generic empathetic responses, yet they fail to assess whether the support is genuinely personalized to users' unique psychological profiles and contextual needs. We introduce EmoHarbor, an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. EmoHarbor employs a Chain-of-Agent architecture that decomposes users' internal processes into three specialized roles, enabling agents to interact with supporters and complete assessments in a manner similar to human users. We instantiate this benchmark using 100 real-world user profiles that cover a diverse range of personality traits and situations, and define 10 evaluation dimensions of personalized support quality. Comprehensive evaluation of 20 advanced LLMs on EmoHarbor reveals a critical insight: while these models excel at generating empathetic responses, they consistently fail to tailor support to individual user contexts. This finding reframes the central challenge, shifting research focus from merely enhancing generic empathy to developing truly user-aware emotional support. EmoHarbor provides a reproducible and scalable framework to guide the development and evaluation of more nuanced and user-aware emotional support systems.