Watching Movies Like a Human: Egocentric Emotion Understanding for Embodied Companions

TL;DR

This paper introduces EgoScreen-Emotion, a new egocentric movie emotion dataset, and a multimodal reasoning framework to improve emotion understanding in realistic viewing scenarios, addressing domain gaps from cinematic footage.

Contribution

The paper presents the first egocentric screen-view movie emotion dataset and a multimodal long-context reasoning model to enhance cross-domain emotion understanding.

Findings

Models trained on cinematic footage perform poorly on egocentric data, with Macro-F1 dropping from 27.99 to 16.69.

Training on ESE improves model robustness in realistic egocentric viewing conditions.

The proposed approach achieves competitive results with strong closed-source models.

Abstract

Embodied robotic agents often perceive movies through an egocentric screen-view interface rather than native cinematic footage, introducing domain shifts such as viewpoint distortion, scale variation, illumination changes, and environmental interference. However, existing research on movie emotion understanding is almost exclusively conducted on cinematic footage, limiting cross-domain generalization to real-world viewing scenarios. To bridge this gap, we introduce EgoScreen-Emotion (ESE), the first benchmark dataset for egocentric screen-view movie emotion understanding. ESE contains 224 movie trailers captured under controlled egocentric screen-view conditions, producing 28,667 temporally aligned key-frames annotated by multiple raters with a confidence-aware multi-label protocol to address emotional ambiguity. We further build a multimodal long-context emotion reasoning framework that models temporal visual evidence, narrative summaries, compressed historical context, and audio cues. Cross-domain experiments reveal a severe domain gap: models trained on cinematic footage drop from 27.99 to 16.69 Macro-F1 when evaluated on realistic egocentric screen-view observations. Training on ESE substantially improves robustness under realistic viewing conditions. Our approach achieves competitive performance compared with strong closed-source multimodal models, highlighting the importance of domain-specific data and long-context multimodal reasoning.