CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generation

TL;DR

CineScene introduces an implicit 3D-aware scene representation framework for cinematic video generation, enabling scene consistency and camera control in synthesized videos from static environment images.

Contribution

The paper proposes a novel implicit 3D scene representation and context conditioning mechanism for cinematic video synthesis, along with a new dataset for training and evaluation.

Findings

Achieves state-of-the-art scene consistency in generated videos

Handles large camera movements effectively

Generalizes across diverse environments

Abstract

Cinematic video production requires control over scene-subject composition and camera movement, but live-action shooting remains costly due to the need for constructing physical sets. To address this, we introduce the task of cinematic video generation with decoupled scene context: given multiple images of a static environment, the goal is to synthesize high-quality videos featuring dynamic subject while preserving the underlying scene consistency and following a user-specified camera trajectory. We present CineScene, a framework that leverages implicit 3D-aware scene representation for cinematic video generation. Our key innovation is a novel context conditioning mechanism that injects 3D-aware features in an implicit way: By encoding scene images into visual representations through VGGT, CineScene injects spatial priors into a pretrained text-to-video generation model by additional context concatenation, enabling camera-controlled video synthesis with consistent scenes and dynamic subjects. To further enhance the model's robustness, we introduce a simple yet effective random-shuffling strategy for the input scene images during training. To address the lack of training data, we construct a scene-decoupled dataset with Unreal Engine 5, containing paired videos of scenes with and without dynamic subjects, panoramic images representing the underlying static scene, along with their camera trajectories. Experiments show that CineScene achieves state-of-the-art performance in scene-consistent cinematic video generation, handling large camera movements and demonstrating generalization across diverse environments.