MoVideo: Motion-Aware Video Generation with Diffusion Models

TL;DR

MoVideo introduces a motion-aware diffusion framework for video generation that explicitly models motion through depth and optical flow, leading to improved temporal consistency and visual quality in generated videos.

Contribution

This work presents a novel diffusion-based video generation method that incorporates motion modeling via depth and optical flow, enhancing temporal coherence and detail preservation.

Findings

Achieves state-of-the-art results in text-to-video generation.

Demonstrates improved prompt and frame consistency.

Produces higher visual quality videos.

Abstract

While recent years have witnessed great progress on using diffusion models for video generation, most of them are simple extensions of image generation frameworks, which fail to explicitly consider one of the key differences between videos and images, i.e., motion. In this paper, we propose a novel motion-aware video generation (MoVideo) framework that takes motion into consideration from two aspects: video depth and optical flow. The former regulates motion by per-frame object distances and spatial layouts, while the later describes motion by cross-frame correspondences that help in preserving fine details and improving temporal consistency. More specifically, given a key frame that exists or generated from text prompts, we first design a diffusion model with spatio-temporal modules to generate the video depth and the corresponding optical flows. Then, the video is generated in the latent space by another spatio-temporal diffusion model under the guidance of depth, optical flow-based warped latent video and the calculated occlusion mask. Lastly, we use optical flows again to align and refine different frames for better video decoding from the latent space to the pixel space. In experiments, MoVideo achieves state-of-the-art results in both text-to-video and image-to-video generation, showing promising prompt consistency, frame consistency and visual quality.