EmpathicStories++: A Multimodal Dataset for Empathy towards Personal Experiences
Jocelyn Shen, Yubin Kim, Mohit Hulse, Wazeer Zulfikar, Sharifa, Alghowinem, Cynthia Breazeal, Hae Won Park

TL;DR
This paper introduces EmpathicStories++, a pioneering multimodal, longitudinal dataset capturing human empathy during personal storytelling, and benchmarks models for predicting empathy in real-world, natural interactions with AI.
Contribution
The paper presents the first longitudinal empathy dataset with multimodal data collected over a month in natural settings, and introduces a novel empathy prediction task with baseline benchmarks.
Findings
EmpathicStories++ contains 53 hours of multimodal data from 41 participants.
State-of-the-art models achieve baseline performance on empathy prediction.
The dataset enables research on real-world, longitudinal empathy modeling.
Abstract
Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset (https://mitmedialab.github.io/empathic-stories-multimodal/) containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants' homes, as participants engage in natural, empathic storytelling…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAI in Service Interactions
