EmoBench: Evaluating the Emotional Intelligence of Large Language Models

TL;DR

EmoBench is a comprehensive benchmark designed to evaluate the emotional intelligence of large language models, addressing limitations of previous datasets by including diverse, psychologically grounded questions in multiple languages.

Contribution

This work introduces EmoBench, a new benchmark with 400 carefully crafted questions based on psychological theories, covering emotional understanding and application for LLM evaluation.

Findings

Existing LLMs show a significant gap in emotional intelligence compared to humans.

EmoBench reveals the need for improved models in emotion regulation and understanding.

The benchmark provides a reliable tool for future EI research in LLMs.

Abstract

Recent advances in Large Language Models (LLMs) have highlighted the need for robust, comprehensive, and challenging benchmarks. Yet, research on evaluating their Emotional Intelligence (EI) is considerably limited. Existing benchmarks have two major shortcomings: first, they mainly focus on emotion recognition, neglecting essential EI capabilities such as emotion regulation and thought facilitation through emotion understanding; second, they are primarily constructed from existing datasets, which include frequent patterns, explicit information, and annotation errors, leading to unreliable evaluation. We propose EmoBench, a benchmark that draws upon established psychological theories and proposes a comprehensive definition for machine EI, including Emotional Understanding and Emotional Application. EmoBench includes a set of 400 hand-crafted questions in English and Chinese, which are meticulously designed to require thorough reasoning and understanding. Our findings reveal a considerable gap between the EI of existing LLMs and the average human, highlighting a promising direction for future research. Our code and data are publicly available at https://github.com/Sahandfer/EmoBench.