Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweening

TL;DR

This paper introduces Motion Prior Distillation (MPD), an inference-time technique that improves temporal coherence in image-to-video diffusion models for inbetweening by reducing path mismatch and artifacts.

Contribution

The paper proposes MPD, a novel distillation method that aligns forward and backward motion priors during inference, enhancing temporal consistency in generated video frames.

Findings

MPD significantly reduces temporal artifacts in generated videos.

Quantitative results show improved coherence on standard benchmarks.

User studies confirm higher perceived quality of inbetweened frames.

Abstract

Recent progress in image-to-video (I2V) diffusion models has significantly advanced the field of generative inbetweening, which aims to generate semantically plausible frames between two keyframes. In particular, inference-time sampling strategies, which leverage the generative priors of large-scale pre-trained I2V models without additional training, have become increasingly popular. However, existing inference-time sampling, either fusing forward and backward paths in parallel or alternating them sequentially, often suffers from temporal discontinuities and undesirable visual artifacts due to the misalignment between the two generated paths. This is because each path follows the motion prior induced by its own conditioning frame. In this work, we propose Motion Prior Distillation (MPD), a simple yet effective inference-time distillation technique that suppresses bidirectional mismatch by distilling the motion residual of the forward path into the backward path. Our method can deliberately avoid denoising the end-conditioned path which causes the ambiguity of the path, and yield more temporally coherent inbetweening results with the forward motion prior. We not only perform quantitative evaluations on standard benchmarks, but also conduct extensive user studies to demonstrate the effectiveness of our approach in practical scenarios.