GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

TL;DR

GimbalDiffusion introduces a gravity-based, absolute coordinate system for precise, interpretable camera control in text-to-video generation, enabling extreme viewpoints and improved guidance.

Contribution

The paper presents GimbalDiffusion, a novel framework that uses gravity as a global reference for absolute camera trajectories, supporting extreme angles and better control.

Findings

Supports full sphere viewpoints including extreme pitch and roll.

Uses null-pitch conditioning to prevent conflicting prompt content.

Proposes new benchmarks for gravity-aware camera control evaluation.

Abstract

Recent progress in text-to-video generation has achieved remarkable realism, yet fine-grained control over camera motion and orientation remains elusive, especially with extreme trajectories (e.g., a 180-degree turnaround, or looking directly up or down). Existing approaches typically encode camera trajectories using relative or ambiguous representations, limiting precise geometric control and offering limited support for large rotations. We introduce GimbalDiffusion, a framework that enables camera control grounded in physical-world coordinates, using gravity as a global reference. Instead of describing motion relative to previous frames, our method defines camera trajectories in an absolute coordinate system, allowing accurate, interpretable control over camera parameters. Using panoramic 360-degree videos for training, we cover the full sphere of possible viewpoints, including combinations of extreme pitch and roll that are out-of-distribution of conventional video data. To improve camera guidance, we introduce null-pitch conditioning, a strategy that prevents the model from overriding camera specifications in the presence of conflicting prompt content (e.g., generating grass while the camera points toward the sky). Finally, we propose new benchmarks to evaluate gravity-aware camera-controlled video generation, assessing models' ability to generate extreme camera angles and quantify their input prompt entanglement.