Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories

TL;DR

This paper introduces Rays as Pixels, a unified Video Diffusion Model that jointly learns to predict camera trajectories and generate videos, improving tasks like pose estimation and camera-controlled video synthesis.

Contribution

It is the first model to unify camera pose prediction and video generation in a single framework using dense ray pixels and a novel attention mechanism.

Findings

Model accurately predicts camera trajectories from videos.

Enables high-quality camera-controlled video generation.

Self-consistency tests confirm reliable pose and rendering predictions.

Abstract

Recovering camera parameters from images and rendering scenes from novel viewpoints have been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task depends on what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. To our knowledge, this is the first model to predict camera poses and do camera-controlled video generation within a single framework. We represent each camera as dense ray pixels (raxels), a pixel-aligned encoding that lives in the same latent space as video frames, and denoise the two jointly through a Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, generating video from input images along a pre-defined trajectory, and jointly synthesizing video and trajectory from input images. We evaluate on pose estimation and camera-controlled video generation, and introduce a closed-loop self-consistency test showing that the model's predicted poses and its renderings conditioned on those poses agree. Ablations against Pl\"ucker embeddings confirm that representing cameras in a shared latent space with video is subtantially more effective.