Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation

TL;DR

Tune-A-Video introduces a one-shot tuning approach for text-to-video generation using pre-trained image diffusion models, enabling efficient and consistent video synthesis from a single text-video pair.

Contribution

The paper proposes a novel one-shot tuning method for text-to-video generation that leverages pre-trained image diffusion models with a new spatio-temporal attention mechanism.

Findings

Effective generation of videos from a single text-video pair

High content consistency in generated videos

Significant reduction in computational cost

Abstract

To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator. Despite their promising results, such paradigm is computationally expensive. In this work, we propose a new T2V generation setting$\unicode{x2014}$One-Shot Video Tuning, where only one text-video pair is presented. Our model is built on state-of-the-art T2I diffusion models pre-trained on massive image data. We make two key observations: 1) T2I models can generate still images that represent verb terms; 2) extending T2I models to generate multiple images concurrently exhibits surprisingly good content consistency. To further learn continuous motion, we introduce Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy. At inference, we employ DDIM inversion to provide structure guidance for sampling. Extensive qualitative and numerical experiments demonstrate the remarkable ability of our method across various applications.