ReRoPE: Repurposing RoPE for Relative Camera Control

TL;DR

ReRoPE introduces a plug-and-play method to incorporate relative camera pose control into pre-trained video diffusion models by leveraging underutilized spectral components of Rotary Positional Embeddings, enhancing control without retraining.

Contribution

It proposes ReRoPE, a novel approach that injects relative camera information into existing models by exploiting spectral bandwidth, enabling controllable video generation efficiently.

Findings

ReRoPE improves camera control accuracy in video generation.

The method maintains high visual fidelity and pre-trained priors.

ReRoPE is training-efficient and adaptable to existing models.

Abstract

Video generation with controllable camera viewpoints is essential for applications such as interactive content creation, gaming, and simulation. Existing methods typically adapt pre-trained video models using camera poses relative to a fixed reference, e.g., the first frame. However, these encodings lack shift-invariance, often leading to poor generalization and accumulated drift. While relative camera pose embeddings defined between arbitrary view pairs offer a more robust alternative, integrating them into pre-trained video diffusion models without prohibitive training costs or architectural changes remains challenging. We introduce ReRoPE, a plug-and-play framework that incorporates relative camera information into pre-trained video diffusion models without compromising their generation capability. Our approach is based on the insight that Rotary Positional Embeddings (RoPE) in existing models underutilize their full spectral bandwidth, particularly in the low-frequency components. By seamlessly injecting relative camera pose information into these underutilized bands, ReRoPE achieves precise control while preserving strong pre-trained generative priors. We evaluate our method on both image-to-video (I2V) and video-to-video (V2V) tasks in terms of camera control accuracy and visual fidelity. Our results demonstrate that ReRoPE offers a training-efficient path toward controllable, high-fidelity video generation. See project page for more results: https://sisyphe-lee.github.io/ReRoPE/