H2HTalk: Evaluating Large Language Models as Emotional Companion

TL;DR

H2HTalk is a comprehensive benchmark designed to evaluate large language models as emotional companions, focusing on empathy, personality development, and safety in support conversations, with extensive scenarios and a new attachment-based safety module.

Contribution

This paper introduces H2HTalk, the first large-scale benchmark for assessing LLMs as emotional companions, including a novel attachment-theory-based safety module and diverse real-world scenarios.

Findings

Models struggle with implicit user needs and evolving conversations.

Long-horizon planning and memory retention are key challenges.

H2HTalk provides a new standard for evaluating emotionally intelligent LLMs.

Abstract

As digital emotional support needs grow, Large Language Model companions offer promising authentic, always-available empathy, though rigorous evaluation lags behind model advancement. We present Heart-to-Heart Talk (H2HTalk), a benchmark assessing companions across personality development and empathetic interaction, balancing emotional intelligence with linguistic fluency. H2HTalk features 4,650 curated scenarios spanning dialogue, recollection, and itinerary planning that mirror real-world support conversations, substantially exceeding previous datasets in scale and diversity. We incorporate a Secure Attachment Persona (SAP) module implementing attachment-theory principles for safer interactions. Benchmarking 50 LLMs with our unified protocol reveals that long-horizon planning and memory retention remain key challenges, with models struggling when user needs are implicit or evolve mid-conversation. H2HTalk establishes the first comprehensive benchmark for emotionally intelligent companions. We release all materials to advance development of LLMs capable of providing meaningful and safe psychological support.