PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth

TL;DR

PosePilot is a novel framework that improves camera pose control in generative world models for autonomous driving by leveraging self-supervised depth estimation and structure-from-motion principles, resulting in more accurate and consistent viewpoint synthesis.

Contribution

It introduces PosePilot, a lightweight method that enhances camera pose controllability in generative models using self-supervised depth and pose estimation techniques.

Findings

Significantly improves structural understanding and motion reasoning.

Enhances viewpoint accuracy and geometric consistency.

Sets new benchmarks for pose controllability in generative models.

Abstract

Recent advancements in autonomous driving (AD) systems have highlighted the potential of world models in achieving robust and generalizable performance across both ordinary and challenging driving conditions. However, a key challenge remains: precise and flexible camera pose control, which is crucial for accurate viewpoint transformation and realistic simulation of scene dynamics. In this paper, we introduce PosePilot, a lightweight yet powerful framework that significantly enhances camera pose controllability in generative world models. Drawing inspiration from self-supervised depth estimation, PosePilot leverages structure-from-motion principles to establish a tight coupling between camera pose and video generation. Specifically, we incorporate self-supervised depth and pose readouts, allowing the model to infer depth and relative camera motion directly from video sequences. These outputs drive pose-aware frame warping, guided by a photometric warping loss that enforces geometric consistency across synthesized frames. To further refine camera pose estimation, we introduce a reverse warping step and a pose regression loss, improving viewpoint precision and adaptability. Extensive experiments on autonomous driving and general-domain video datasets demonstrate that PosePilot significantly enhances structural understanding and motion reasoning in both diffusion-based and auto-regressive world models. By steering camera pose with self-supervised depth, PosePilot sets a new benchmark for pose controllability, enabling physically consistent, reliable viewpoint synthesis in generative world models.