I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models

TL;DR

I2V-Adapter is a versatile, lightweight module that enhances image-to-video generation by preserving input image identity and maintaining compatibility with existing models, enabling high-quality, controllable videos.

Contribution

It introduces a novel cross-frame attention mechanism and a Frame Similarity Prior, allowing effective image-to-video translation without altering pretrained models.

Findings

Produces high-quality, coherent videos with preserved image identity

Requires only a few trainable parameters, reducing training costs

Ensures compatibility with existing community-driven models

Abstract

Text-guided image-to-video (I2V) generation aims to generate a coherent video that preserves the identity of the input image and semantically aligns with the input prompt. Existing methods typically augment pretrained text-to-video (T2V) models by either concatenating the image with noised video frames channel-wise before being fed into the model or injecting the image embedding produced by pretrained image encoders in cross-attention modules. However, the former approach often necessitates altering the fundamental weights of pretrained T2V models, thus restricting the model's compatibility within the open-source communities and disrupting the model's prior knowledge. Meanwhile, the latter typically fails to preserve the identity of the input image. We present I2V-Adapter to overcome such limitations. I2V-Adapter adeptly propagates the unnoised input image to subsequent noised frames through a cross-frame attention mechanism, maintaining the identity of the input image without any changes to the pretrained T2V model. Notably, I2V-Adapter only introduces a few trainable parameters, significantly alleviating the training cost and also ensures compatibility with existing community-driven personalized models and control tools. Moreover, we propose a novel Frame Similarity Prior to balance the motion amplitude and the stability of generated videos through two adjustable control coefficients. Our experimental results demonstrate that I2V-Adapter is capable of producing high-quality videos. This performance, coupled with its agility and adaptability, represents a substantial advancement in the field of I2V, particularly for personalized and controllable applications.