DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization

TL;DR

This paper introduces a distillation and latent reward optimization method for efficient few-step video generation, achieving high quality and diversity with significantly reduced computation.

Contribution

It presents a novel distillation technique combining variational score and consistency distillation for fast video generation, plus a latent reward model for performance enhancement.

Findings

Achieves state-of-the-art results in few-step video generation (10 seconds, 128 frames)

Distilled models outperform baseline models and the original teacher model

One-step distillation accelerates sampling by up to 278.6 times

Abstract

Diffusion probabilistic models have shown significant progress in video generation; however, their computational efficiency is limited by the large number of sampling steps required. Reducing sampling steps often compromises video quality or generation diversity. In this work, we introduce a distillation method that combines variational score distillation and consistency distillation to achieve few-step video generation, maintaining both high quality and diversity. We also propose a latent reward model fine-tuning approach to further enhance video generation performance according to any specified reward metric. This approach reduces memory usage and does not require the reward to be differentiable. Our method demonstrates state-of-the-art performance in few-step generation for 10-second videos (128 frames at 12 FPS). The distilled student model achieves a score of 82.57 on VBench, surpassing the teacher model as well as baseline models Gen-3, T2V-Turbo, and Kling. One-step distillation accelerates the teacher model's diffusion sampling by up to 278.6 times, enabling near real-time generation. Human evaluations further validate the superior performance of our 4-step student models compared to teacher model using 50-step DDIM sampling.