BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations

TL;DR

BlobGEN-Vid introduces a controllable, compositional text-to-video generation framework using blob video representations, enabling detailed object control and smooth transitions, outperforming existing models in zero-shot and layout controllability tasks.

Contribution

We propose a novel blob video representation and a diffusion model that enhances controllability and compositionality in text-to-video generation, with effective regional consistency and semantic interpolation.

Findings

Achieves state-of-the-art zero-shot video generation.

Outperforms existing models in layout controllability.

Enables fine-grained control over object motion and appearance.

Abstract

Existing video generation models struggle to follow complex text prompts and synthesize multiple objects, raising the need for additional grounding input for improved controllability. In this work, we propose to decompose videos into visual primitives - blob video representation, a general representation for controllable video generation. Based on blob conditions, we develop a blob-grounded video diffusion model named BlobGEN-Vid that allows users to control object motions and fine-grained object appearance. In particular, we introduce a masked 3D attention module that effectively improves regional consistency across frames. In addition, we introduce a learnable module to interpolate text embeddings so that users can control semantics in specific frames and obtain smooth object transitions. We show that our framework is model-agnostic and build BlobGEN-Vid based on both U-Net and DiT-based video diffusion models. Extensive experimental results show that BlobGEN-Vid achieves superior zero-shot video generation ability and state-of-the-art layout controllability on multiple benchmarks. When combined with an LLM for layout planning, our framework even outperforms proprietary text-to-video generators in terms of compositional accuracy.