MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion

TL;DR

MotionAura introduces a novel framework combining vector-quantized diffusion models, spectral transformers, and a new VAE for high-quality, motion-consistent text-to-video generation and inpainting, advancing spatiotemporal video processing.

Contribution

The paper presents a new 3D VAE for video compression, a vector-quantized diffusion model for text-to-video synthesis, and a spectral transformer for improved denoising, along with a sketch-guided inpainting task.

Findings

Achieves state-of-the-art reconstruction quality.

Produces temporally coherent videos aligned with text prompts.

Demonstrates superior performance on multiple benchmarks.

Abstract

The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Vector-Quantization Variational Autoencoder (3D-MBQ-VAE), which combines Variational Autoencoders (VAEs) with masked token modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing a novel training strategy with full frame masking. Second, we present MotionAura, a text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture complex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, we propose a spectral transformer-based denoising network that processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-quality video generation and denoising. Lastly, we introduce a downstream task of Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our models achieve SOTA performance on a range of benchmarks. Our work offers robust frameworks for spatiotemporal modeling and user-driven video content manipulation. We will release the code, datasets, and models in open-source.