InstanceV: Instance-Level Video Generation

TL;DR

InstanceV introduces an instance-level controllable video generation framework that enhances fine-grained control and semantic consistency, outperforming existing models in quality and instance accuracy.

Contribution

The paper presents InstanceV, a novel framework with instance-aware mechanisms and a new benchmark for improved fine-grained control in video generation.

Findings

Achieves superior instance-level controllability in video synthesis.

Outperforms state-of-the-art models in quality and instance metrics.

Introduces InstanceBench for comprehensive evaluation.

Abstract

Recent advances in text-to-video diffusion models have enabled the generation of high-quality videos conditioned on textual descriptions. However, most existing text-to-video models rely solely on textual conditions, lacking general fine-grained controllability over video generation. To address this challenge, we propose InstanceV, a video generation framework that enables i) instance-level control and ii) global semantic consistency. Specifically, with the aid of proposed Instance-aware Masked Cross-Attention mechanism, InstanceV maximizes the utilization of additional instance-level grounding information to generate correctly attributed instances at designated spatial locations. To improve overall consistency, We introduce the Shared Timestep-Adaptive Prompt Enhancement module, which connects local instances with global semantics in a parameter-efficient manner. Furthermore, we incorporate Spatially-Aware Unconditional Guidance during both training and inference to alleviate the disappearance of small instances. Finally, we propose a new benchmark, named InstanceBench, which combines general video quality metrics with instance-aware metrics for more comprehensive evaluation on instance-level video generation. Extensive experiments demonstrate that InstanceV not only achieves remarkable instance-level controllability in video generation, but also outperforms existing state-of-the-art models in both general quality and instance-aware metrics across qualitative and quantitative evaluations.