OnlyFlow: Optical Flow based Motion Conditioning for Video Diffusion Models

TL;DR

OnlyFlow introduces a novel optical flow-based method for controlling motion in text-to-video generation, enabling precise, user-guided video synthesis without extensive retraining.

Contribution

It leverages optical flow from input videos to condition motion in generated videos, offering a versatile and lightweight control mechanism for text-to-video models.

Findings

Outperforms state-of-the-art methods in various tasks

Works effectively without task-specific training

Provides precise motion control based on input video flow

Abstract

We consider the problem of text-to-video generation tasks with precise control for various applications such as camera movement control and video-to-video editing. Most methods tacking this problem rely on providing user-defined controls, such as binary masks or camera movement embeddings. In our approach we propose OnlyFlow, an approach leveraging the optical flow firstly extracted from an input video to condition the motion of generated videos. Using a text prompt and an input video, OnlyFlow allows the user to generate videos that respect the motion of the input video as well as the text prompt. This is implemented through an optical flow estimation model applied on the input video, which is then fed to a trainable optical flow encoder. The output feature maps are then injected into the text-to-video backbone model. We perform quantitative, qualitative and user preference studies to show that OnlyFlow positively compares to state-of-the-art methods on a wide range of tasks, even though OnlyFlow was not specifically trained for such tasks. OnlyFlow thus constitutes a versatile, lightweight yet efficient method for controlling motion in text-to-video generation. Models and code will be made available on GitHub and HuggingFace.