ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encoding

TL;DR

ReDirector introduces Rotary Camera Encoding to generate customizable, length-variable video retakes with improved camera control, geometric consistency, and background preservation, advancing dynamic video editing capabilities.

Contribution

The paper proposes Rotary Camera Encoding (RoCE), a novel camera-conditioned RoPE phase shift that enhances video retake generation by capturing multi-view relationships and generalizing to diverse camera trajectories.

Findings

Improved camera controllability and geometric consistency

Enhanced background preservation in retake videos

Significant quality improvements across various trajectories and lengths

Abstract

We present ReDirector, a novel camera-controlled video retake generation method for dynamically captured variable-length videos. In particular, we rectify a common misuse of RoPE in previous works by aligning the spatiotemporal positions of the input video and the target retake. Moreover, we introduce Rotary Camera Encoding (RoCE), a camera-conditioned RoPE phase shift that captures and integrates multi-view relationships within and across the input and target videos. By integrating camera conditions into RoPE, our method generalizes to out-of-distribution camera trajectories and video lengths, yielding improved dynamic object localization and static background preservation. Extensive experiments further demonstrate significant improvements in camera controllability, geometric consistency, and video quality across various trajectories and lengths.