MOVi: Training-free Text-conditioned Multi-Object Video Generation

TL;DR

This paper introduces a training-free method for multi-object text-conditioned video generation that improves object interaction accuracy and motion realism by leveraging large language models and attention manipulation.

Contribution

It presents a novel training-free approach that uses LLMs as object trajectory directors and refines attention mechanisms to enhance multi-object video generation.

Findings

42% improvement in motion dynamics and object accuracy

Enhanced object-specific feature capture and motion patterns

Maintained high fidelity and smoothness in generated videos

Abstract

Recent advances in diffusion-based text-to-video (T2V) models have demonstrated remarkable progress, but these models still face challenges in generating videos with multiple objects. Most models struggle with accurately capturing complex object interactions, often treating some objects as static background elements and limiting their movement. In addition, they often fail to generate multiple distinct objects as specified in the prompt, resulting in incorrect generations or mixed features across objects. In this paper, we present a novel training-free approach for multi-object video generation that leverages the open world knowledge of diffusion models and large language models (LLMs). We use an LLM as the ``director'' of object trajectories, and apply the trajectories through noise re-initialization to achieve precise control of realistic movements. We further refine the generation process by manipulating the attention mechanism to better capture object-specific features and motion patterns, and prevent cross-object feature interference. Extensive experiments validate the effectiveness of our training free approach in significantly enhancing the multi-object generation capabilities of existing video diffusion models, resulting in 42% absolute improvement in motion dynamics and object generation accuracy, while also maintaining high fidelity and motion smoothness.