Cross-View World Models

TL;DR

This paper introduces Cross-View World Models (XVWM) that learn view-invariant environment representations by predicting future states across different viewpoints, enhancing planning and spatial understanding in agents.

Contribution

The paper presents a novel cross-view prediction objective for training world models, enabling multi-view consistency and improved spatial reasoning in agents.

Findings

Multi-view consistency improves spatially grounded representations.

The model enables planning from multiple viewpoints, including egocentric and bird's-eye views.

Predicting from different perspectives may facilitate perspective-taking in multi-agent systems.

Abstract

World models enable agents to plan by imagining future states, but existing approaches operate from a single viewpoint, typically egocentric, even when other perspectives would make planning easier; navigation, for instance, benefits from a bird's-eye view. We introduce Cross-View World Models (XVWM), trained with a cross-view prediction objective: given a sequence of frames from one viewpoint, predict the future state from the same or a different viewpoint after an action is taken. Enforcing cross-view consistency acts as geometric regularization: because the input and output views may share little or no visual overlap, to predict across viewpoints, the model must learn view-invariant representations of the environment's 3D structure. We train on synchronized multi-view gameplay data from Aimlabs, an aim-training platform providing precisely aligned multi-camera recordings with high-frequency action labels. The resulting model gives agents parallel imagination streams across viewpoints, enabling planning in whichever frame of reference best suits the task while executing from the egocentric view. Our results show that multi-view consistency provides a strong learning signal for spatially grounded representations. Finally, predicting the consequences of one's actions from another viewpoint may offer a foundation for perspective-taking in multi-agent settings.