PlayerOne: Egocentric World Simulator

TL;DR

PlayerOne is a pioneering egocentric world simulator that creates realistic, controllable videos aligned with user motion, enabling immersive exploration and scene modeling in dynamic environments.

Contribution

It introduces the first egocentric realistic world simulator with a novel coarse-to-fine training pipeline and part-disentangled motion control for precise scene and motion modeling.

Findings

Demonstrates strong generalization in human movement control.

Achieves accurate long-form scene and video frame reconstruction.

Enables diverse scenario modeling with high scene consistency.

Abstract

We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and worldconsistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications.