MemEmo: Evaluating Emotion in Memory Systems of Agents

TL;DR

This paper introduces MemEmo, a benchmark for evaluating emotion processing in memory systems of AI agents, revealing current limitations and guiding future improvements.

Contribution

It presents the HLME dataset and a comprehensive evaluation framework for assessing emotion-related memory tasks in AI systems, highlighting their deficiencies.

Findings

None of the systems perform well across all tasks.

Current memory systems struggle with emotional information processing.

The benchmark reveals significant gaps in emotion handling capabilities.

Abstract

Memory systems address the challenge of context loss in Large Language Model during prolonged interactions. However, compared to human cognition, the efficacy of these systems in processing emotion-related information remains inconclusive. To address this gap, we propose an emotion-enhanced memory evaluation benchmark to assess the performance of mainstream and state-of-the-art memory systems in handling affective information. We developed the \textbf{H}uman-\textbf{L}ike \textbf{M}emory \textbf{E}motion (\textbf{HLME}) dataset, which evaluates memory systems across three dimensions: emotional information extraction, emotional memory updating, and emotional memory question answering. Experimental results indicate that none of the evaluated systems achieve robust performance across all three tasks. Our findings provide an objective perspective on the current deficiencies of memory systems in processing emotional memories and suggest a new trajectory for future research and system optimization.