AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction

TL;DR

This paper introduces AID, a novel method that adapts Image2Video diffusion models with instruction-guided control for improved video prediction, achieving state-of-the-art results across multiple datasets.

Contribution

The paper proposes a new framework combining a dual query transformer and adapters to transfer pretrained Image2Video models for instruction-guided video prediction with minimal training.

Findings

Significant improvements in FVD scores on multiple datasets.

Outperforms existing state-of-the-art methods in instruction-guided video prediction.

Demonstrates effective transfer of video dynamic priors with minimal training.

Abstract

Text-guided video prediction (TVP) involves predicting the motion of future frames from the initial frame according to an instruction, which has wide applications in virtual reality, robotics, and content creation. Previous TVP methods make significant breakthroughs by adapting Stable Diffusion for this task. However, they struggle with frame consistency and temporal stability primarily due to the limited scale of video datasets. We observe that pretrained Image2Video diffusion models possess good priors for video dynamics but they lack textual control. Hence, transferring Image2Video models to leverage their video dynamic priors while injecting instruction control to generate controllable videos is both a meaningful and challenging task. To achieve this, we introduce the Multi-Modal Large Language Model (MLLM) to predict future video states based on initial frames and text instructions. More specifically, we design a dual query transformer (DQFormer) architecture, which integrates the instructions and frames into the conditional embeddings for future frame prediction. Additionally, we develop Long-Short Term Temporal Adapters and Spatial Adapters that can quickly transfer general video diffusion models to specific scenarios with minimal training costs. Experimental results show that our method significantly outperforms state-of-the-art techniques on four datasets: Something Something V2, Epic Kitchen-100, Bridge Data, and UCF-101. Notably, AID achieves 91.2% and 55.5% FVD improvements on Bridge and SSv2 respectively, demonstrating its effectiveness in various domains. More examples can be found at our website https://chenhsing.github.io/AID.