Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model

TL;DR

This paper identifies the problem of conditional image leakage in image-to-video diffusion models, which causes less motion in generated videos, and proposes methods to mitigate this issue for improved motion realism and consistency.

Contribution

The paper introduces novel inference and training strategies, including early start generation and time-dependent noise, to reduce image leakage and enhance motion in video generation.

Findings

Outperforms baselines in motion scores and error metrics

Maintains image alignment and temporal consistency

Enables more accurate motion control

Abstract

Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-video diffusion models (I2V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training, applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency on it. We validate these general strategies on various I2V-DMs on our collected open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: \url{https://cond-image-leak.github.io/}.