OmniDrag: Enabling Motion Control for Omnidirectional Image-to-Video Generation

TL;DR

OmniDrag introduces a novel method for controllable omnidirectional video generation that enables precise scene- and object-level motion control, overcoming previous limitations in content accuracy and complex spherical motions.

Contribution

It is the first approach to jointly enable scene- and object-level motion control for high-quality omnidirectional image-to-video generation, utilizing a new spherical motion estimator and dataset.

Findings

Outperforms existing methods in control accuracy and video quality.

Effectively handles complex spherical motions with minimal spatial distortion.

Provides a user-friendly drag-style control interface.

Abstract

As virtual reality gains popularity, the demand for controllable creation of immersive and dynamic omnidirectional videos (ODVs) is increasing. While previous text-to-ODV generation methods achieve impressive results, they struggle with content inaccuracies and inconsistencies due to reliance solely on textual inputs. Although recent motion control techniques provide fine-grained control for video generation, directly applying these methods to ODVs often results in spatial distortion and unsatisfactory performance, especially with complex spherical motions. To tackle these challenges, we propose OmniDrag, the first approach enabling both scene- and object-level motion control for accurate, high-quality omnidirectional image-to-video generation. Building on pretrained video diffusion models, we introduce an omnidirectional control module, which is jointly fine-tuned with temporal attention layers to effectively handle complex spherical motion. In addition, we develop a novel spherical motion estimator that accurately extracts motion-control signals and allows users to perform drag-style ODV generation by simply drawing handle and target points. We also present a new dataset, named Move360, addressing the scarcity of ODV data with large scene and object motions. Experiments demonstrate the significant superiority of OmniDrag in achieving holistic scene-level and fine-grained object-level control for ODV generation. The project page is available at https://lwq20020127.github.io/OmniDrag.