TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation

TL;DR

TIV-Diffusion is a novel diffusion-based framework for text-driven image-to-video generation that emphasizes object-centric alignment to improve control and quality of generated videos.

Contribution

The paper introduces an object-centric textual-visual alignment module and a fused knowledge approach, enhancing control and quality in text-driven video generation.

Findings

Achieves state-of-the-art video quality in TI2V tasks.

Effectively maintains object consistency and motion alignment.

Reduces object disappearance and misalignment issues.

Abstract

Text-driven Image to Video Generation (TI2V) aims to generate controllable video given the first frame and corresponding textual description. The primary challenges of this task lie in two parts: (i) how to identify the target objects and ensure the consistency between the movement trajectory and the textual description. (ii) how to improve the subjective quality of generated videos. To tackle the above challenges, we propose a new diffusion-based TI2V framework, termed TIV-Diffusion, via object-centric textual-visual alignment, intending to achieve precise control and high-quality video generation based on textual-described motion for different objects. Concretely, we enable our TIV-Diffuion model to perceive the textual-described objects and their motion trajectory by incorporating the fused textual and visual knowledge through scale-offset modulation. Moreover, to mitigate the problems of object disappearance and misaligned objects and motion, we introduce an object-centric textual-visual alignment module, which reduces the risk of misaligned objects/motion by decoupling the objects in the reference image and aligning textual features with each object individually. Based on the above innovations, our TIV-Diffusion achieves state-of-the-art high-quality video generation compared with existing TI2V methods.