M4V: Multi-Modal Mamba for Text-to-Video Generation

TL;DR

M4V introduces a multi-modal, efficient Transformer-based framework for text-to-video generation that reduces computational costs and enhances visual quality through innovative token re-composition and reward learning strategies.

Contribution

The paper presents M4V, a novel multi-modal Mamba architecture that integrates multi-modal information and spatiotemporal modeling, reducing FLOPs by 45% and improving video quality in text-to-video synthesis.

Findings

Reduces FLOPs by 45% at 768×1280 resolution.

Achieves high-quality video generation with lower computational costs.

Enhances visual realism using a reward learning strategy.

Abstract

Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators. However, modeling the vast spatiotemporal space remains computationally demanding, particularly when employing Transformers, which incur quadratic complexity in sequence processing and thus limit practical applications. Recent advancements in linear-time sequence modeling, particularly the Mamba architecture, offer a more efficient alternative. Nevertheless, its plain design limits its direct applicability to multi-modal and spatiotemporal video generation tasks. To address these challenges, we introduce M4V, a Multi-Modal Mamba framework for text-to-video generation. Specifically, we propose a multi-modal diffusion Mamba (MM-DiM) block that enables seamless integration of multi-modal information and spatiotemporal modeling through a multi-modal token re-composition design. As a result, the Mamba blocks in M4V reduce FLOPs by 45% compared to the attention-based alternative when generating videos at 768$\times$1280 resolution. Additionally, to mitigate the visual quality degradation in long-context autoregressive generation processes, we introduce a reward learning strategy that further enhances per-frame visual realism. Extensive experiments on text-to-video benchmarks demonstrate M4V's ability to produce high-quality videos while significantly lowering computational costs. Code and models will be publicly available at https://huangjch526.github.io/M4V_project.