Unified Camera Positional Encoding for Controlled Video Generation

TL;DR

This paper introduces UCPE, a unified camera encoding method that captures complete camera information, enabling advanced controllability in video generation with minimal additional training parameters.

Contribution

We propose Relative Ray Encoding and Absolute Orientation Encoding to create UCPE, a comprehensive camera representation that improves controllability and generalization in transformer-based video generation.

Findings

Achieves state-of-the-art camera controllability and visual fidelity.

Supports diverse camera intrinsics and lens distortions.

Requires less than 1% additional trainable parameters.

Abstract

Transformers have emerged as a universal backbone across 3D perception, video generation, and world models for autonomous driving and embodied AI, where understanding camera geometry is essential for grounding visual observations in three-dimensional space. However, existing camera encoding methods often rely on simplified pinhole assumptions, restricting generalization across the diverse intrinsics and lens distortions in real-world cameras. We introduce Relative Ray Encoding, a geometry-consistent representation that unifies complete camera information, including 6-DoF poses, intrinsics, and lens distortions. To evaluate its capability under diverse controllability demands, we adopt camera-controlled text-to-video generation as a testbed task. Within this setting, we further identify pitch and roll as two components effective for Absolute Orientation Encoding, enabling full control over the initial camera orientation. Together, these designs form UCPE (Unified Camera Positional Encoding), which integrates into a pretrained video Diffusion Transformer through a lightweight spatial attention adapter, adding less than 1% trainable parameters while achieving state-of-the-art camera controllability and visual fidelity. To facilitate systematic training and evaluation, we construct a large video dataset covering a wide range of camera motions and lens types. Extensive experiments validate the effectiveness of UCPE in camera-controllable video generation and highlight its potential as a general camera representation for Transformers across future multi-view, video, and 3D tasks. Code will be available at https://github.com/chengzhag/UCPE.