AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence

TL;DR

AttuneBench is a new benchmark for evaluating emotional intelligence in large language models through real multi-turn conversations with detailed annotations.

Contribution

It introduces a framework for assessing multiple aspects of emotional intelligence in LLMs using genuine multi-turn interactions and turn-by-turn annotations.

Findings

Model rankings vary across emotion recognition and response quality tasks.

Preference prediction and response quality are more discriminative than emotion-label accuracy.

Emotionally intelligent behavior involves predicting user-specific responses in context.

Abstract

Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others' emotional states, is central to human communication, and increasingly important to assess as LLMs assume conversational roles in everyday life. Existing EI benchmarks rely on synthetic prompts, single-turn cases, or third-party annotation. These approaches do not directly measure how models infer and respond to a participant's emotional state over the course of a real conversation. We introduce AttuneBench, a benchmark grounded in 200 genuine multi-turn human-model conversations in which participants conversed with anonymized LLMs and provided turn-by-turn annotations of their emotional state, the model's behavior, and their preferred responses. Across 11 evaluated models, we find that model rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent, indicating that emotionally intelligent behavior decomposes into separable capabilities. Preference alignment and response-quality judgments are substantially more model-discriminating than emotion-label accuracy. These results indicate that emotionally intelligent behavior requires predicting what kind of response a specific user wants in context, a distinction that aggregate scoring can obscure and that single-turn or synthetic formats cannot directly capture across turns. AttuneBench provides a framework for assessing each of these capabilities and for diagnosing model-specific strengths and failure modes in emotionally salient conversation.