Dynamic-I2V: Exploring Image-to-Video Generation Models via Multimodal LLM

TL;DR

Dynamic-I2V introduces a multimodal large language model-based framework that enhances image-to-video generation by improving motion control and temporal coherence, addressing complex scene understanding challenges.

Contribution

It integrates MLLMs with diffusion transformers for better multimodal encoding, and proposes DIVE, a new benchmark for evaluating dynamic quality in I2V generation.

Findings

Achieves 42.5% improvement in dynamic range.

Attains 7.9% better controllability.

Reaches 11.8% higher quality in generated videos.

Abstract

Recent advancements in image-to-video (I2V) generation have shown promising performance in conventional scenarios. However, these methods still encounter significant challenges when dealing with complex scenes that require a deep understanding of nuanced motion and intricate object-action relationships. To address these challenges, we present Dynamic-I2V, an innovative framework that integrates Multimodal Large Language Models (MLLMs) to jointly encode visual and textual conditions for a diffusion transformer (DiT) architecture. By leveraging the advanced multimodal understanding capabilities of MLLMs, our model significantly improves motion controllability and temporal coherence in synthesized videos. The inherent multimodality of Dynamic-I2V further enables flexible support for diverse conditional inputs, extending its applicability to various downstream generation tasks. Through systematic analysis, we identify a critical limitation in current I2V benchmarks: a significant bias towards favoring low-dynamic videos, stemming from an inadequate balance between motion complexity and visual quality metrics. To resolve this evaluation gap, we propose DIVE - a novel assessment benchmark specifically designed for comprehensive dynamic quality measurement in I2V generation. In conclusion, extensive quantitative and qualitative experiments confirm that Dynamic-I2V attains state-of-the-art performance in image-to-video generation, particularly revealing significant improvements of 42.5%, 7.9%, and 11.8% in dynamic range, controllability, and quality, respectively, as assessed by the DIVE metric in comparison to existing methods.