Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning

TL;DR

This paper introduces DEMO, a novel framework that decomposes text encoding and conditioning into content and motion components, significantly improving motion realism in text-to-video generation.

Contribution

The paper proposes a new decomposed encoding and conditioning framework for T2V that explicitly models motion, addressing limitations of previous static or minimally dynamic video outputs.

Findings

DEMO outperforms existing models in motion dynamics on multiple benchmarks.

Enhanced motion understanding leads to more realistic and dynamic videos.

The approach maintains high visual quality while improving motion synthesis.

Abstract

Despite advancements in Text-to-Video (T2V) generation, producing videos with realistic motion remains challenging. Current models often yield static or minimally dynamic outputs, failing to capture complex motions described by text. This issue stems from the internal biases in text encoding, which overlooks motions, and inadequate conditioning mechanisms in T2V generation models. To address this, we propose a novel framework called DEcomposed MOtion (DEMO), which enhances motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components. Our method includes a content encoder for static elements and a motion encoder for temporal dynamics, alongside separate content and motion conditioning mechanisms. Crucially, we introduce text-motion and video-motion supervision to improve the model's understanding and generation of motion. Evaluations on benchmarks such as MSR-VTT, UCF-101, WebVid-10M, EvalCrafter, and VBench demonstrate DEMO's superior ability to produce videos with enhanced motion dynamics while maintaining high visual quality. Our approach significantly advances T2V generation by integrating comprehensive motion understanding directly from textual descriptions. Project page: https://PR-Ryan.github.io/DEMO-project/