EmpathicStories++: A Multimodal Dataset for Empathy towards Personal Experiences

TL;DR

This paper introduces EmpathicStories++, a pioneering multimodal, longitudinal dataset capturing human empathy during personal storytelling, and benchmarks models for predicting empathy in real-world, natural interactions with AI.

Contribution

The paper presents the first longitudinal empathy dataset with multimodal data collected over a month in natural settings, and introduces a novel empathy prediction task with baseline benchmarks.

Findings

EmpathicStories++ contains 53 hours of multimodal data from 41 participants.

State-of-the-art models achieve baseline performance on empathy prediction.

The dataset enables research on real-world, longitudinal empathy modeling.

Abstract

Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset (https://mitmedialab.github.io/empathic-stories-multimodal/) containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants' homes, as participants engage in natural, empathic storytelling interactions with AI agents. We then introduce a novel task of predicting individuals' empathy toward others' stories based on their personal experiences, evaluated in two contexts: participants' own personal shared story context and their reflections on stories they read. We benchmark this task using state-of-the-art models to pave the way for future improvements in contextualized and longitudinal empathy modeling. Our work provides a valuable resource for further research in developing empathetic AI systems and understanding the intricacies of human empathy within genuine, real-world settings.