EgoM2P: Egocentric Multimodal Multitask Pretraining

TL;DR

EgoM2P is a large, efficient, multimodal pretraining framework for egocentric vision that supports multiple perception and synthesis tasks, outperforming specialized models and enabling comprehensive understanding of first-person video data.

Contribution

The paper introduces EgoM2P, a novel masked modeling framework with temporal tokenizers for egocentric multimodal pretraining, supporting diverse tasks with improved speed and performance.

Findings

EgoM2P matches or exceeds specialist models in various tasks.

EgoM2P is an order of magnitude faster than existing models.

The framework effectively supports multitasking in egocentric vision.

Abstract

Understanding multimodal signals in egocentric vision, such as RGB video, depth, camera poses, and gaze, is essential for applications in augmented reality, robotics, and human-computer interaction, enabling systems to better interpret the camera wearer's actions, intentions, and surrounding environment. However, building large-scale egocentric multimodal and multitask models presents unique challenges. Egocentric data are inherently heterogeneous, with large variations in modality coverage across devices and settings. Generating pseudo-labels for missing modalities, such as gaze or head-mounted camera trajectories, is often infeasible, making standard supervised learning approaches difficult to scale. Furthermore, dynamic camera motion and the complex temporal and spatial structure of first-person video pose additional challenges for the direct application of existing multimodal foundation models. To address these challenges, we introduce a set of efficient temporal tokenizers and propose EgoM2P, a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding. This unified design supports multitasking across diverse egocentric perception and synthesis tasks, including gaze prediction, egocentric camera tracking, and monocular depth estimation from egocentric video, and also serves as a generative model for conditional egocentric video synthesis. Across these tasks, EgoM2P matches or outperforms specialist models while being an order of magnitude faster. We will fully open-source EgoM2P to support the community and advance egocentric vision research. Project page: https://egom2p.github.io/.