EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation

TL;DR

EgoReAct introduces a real-time autoregressive framework that generates 3D-aligned human reactions from egocentric videos, addressing data scarcity and spatial misalignment issues with a new dataset and advanced modeling techniques.

Contribution

The paper presents the first autoregressive model for 3D reaction generation from egocentric videos, utilizing a new dataset and incorporating 3D dynamic features for improved realism and spatial consistency.

Findings

EgoReAct outperforms prior methods in realism and spatial consistency.

The model achieves real-time reaction generation with strict causality.

The Human Reaction Dataset (HRD) effectively addresses data scarcity and misalignment.

Abstract

Humans exhibit adaptive, context-sensitive responses to egocentric visual input. However, faithfully modeling such reactions from egocentric video remains challenging due to the dual requirements of strictly causal generation and precise 3D spatial alignment. To tackle this problem, we first construct the Human Reaction Dataset (HRD) to address data scarcity and misalignment by building a spatially aligned egocentric video-reaction dataset, as existing datasets (e.g., ViMo) suffer from significant spatial inconsistency between the egocentric video and reaction motion, e.g., dynamically moving motions are always paired with fixed-camera videos. Leveraging HRD, we present EgoReAct, the first autoregressive framework that generates 3D-aligned human reaction motions from egocentric video streams in real-time. We first compress the reaction motion into a compact yet expressive latent space via a Vector Quantised-Variational AutoEncoder and then train a Generative Pre-trained Transformer for reaction generation from the visual input. EgoReAct incorporates 3D dynamic features, i.e., metric depth, and head dynamics during the generation, which effectively enhance spatial grounding. Extensive experiments demonstrate that EgoReAct achieves remarkably higher realism, spatial consistency, and generation efficiency compared with prior methods, while maintaining strict causality during generation. We will release code, models, and data upon acceptance.