ExVideo: Extending Video Diffusion Models via Parameter-Efficient Post-Tuning

TL;DR

ExVideo introduces a parameter-efficient post-tuning method that significantly extends the length of videos generated by diffusion models like Stable Video Diffusion, with minimal additional training costs.

Contribution

The paper proposes a novel post-tuning approach that enhances existing video diffusion models to generate longer videos without extensive retraining.

Findings

Increased video length by up to 5x with minimal training resources.

Maintains original model's generalization and diversity in generated videos.

Requires only 1.5k GPU hours for extension training on 40k videos.

Abstract

Recently, advancements in video synthesis have attracted significant attention. Video synthesis models such as AnimateDiff and Stable Video Diffusion have demonstrated the practical applicability of diffusion models in creating dynamic visual content. The emergence of SORA has further spotlighted the potential of video generation technologies. Nonetheless, the extension of video lengths has been constrained by the limitations in computational resources. Most existing video synthesis models can only generate short video clips. In this paper, we propose a novel post-tuning methodology for video synthesis models, called ExVideo. This approach is designed to enhance the capability of current video synthesis models, allowing them to produce content over extended temporal durations while incurring lower training expenditures. In particular, we design extension strategies across common temporal model architectures respectively, including 3D convolution, temporal attention, and positional embedding. To evaluate the efficacy of our proposed post-tuning approach, we conduct extension training on the Stable Video Diffusion model. Our approach augments the model's capacity to generate up to $5\times$ its original number of frames, requiring only 1.5k GPU hours of training on a dataset comprising 40k videos. Importantly, the substantial increase in video length doesn't compromise the model's innate generalization capabilities, and the model showcases its advantages in generating videos of diverse styles and resolutions. We will release the source code and the enhanced model publicly.