egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks
Matthias Jammot, Bj\"orn Braun, Paul Streli, Rafael Wampfler, Christian Holz

TL;DR
egoEMOTION introduces a comprehensive egocentric dataset combining visual, physiological, and self-reported emotion and personality data, enabling advanced research in affect-aware perception and behavior modeling in real-world scenarios.
Contribution
This paper presents the first dataset coupling egocentric vision, physiological signals, and self-reports of emotion and personality, along with benchmark tasks for affect and personality recognition.
Findings
Classical learning methods outperform physiological signals in real-world affect prediction.
The dataset enables new research directions in affect-driven behavior modeling.
Benchmark tasks demonstrate the feasibility of emotion and personality inference from egocentric data.
Abstract
Understanding affect is central to anticipating human behavior, yet current egocentric vision benchmarks largely ignore the person's emotional states that shape their decisions and actions. Existing tasks in egocentric perception focus on physical activities, hand-object interactions, and attention modeling - assuming neutral affect and uniform personality. This limits the ability of vision systems to capture key internal drivers of behavior. In this paper, we present egoEMOTION, the first dataset that couples egocentric visual and physiological signals with dense self-reports of emotion and personality across controlled and real-world scenarios. Our dataset includes over 50 hours of recordings from 43 participants, captured using Meta's Project Aria glasses. Each session provides synchronized eye-tracking video, headmounted photoplethysmography, inertial motion data, and physiological…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
