SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces

TL;DR

This paper introduces a novel approach for long-term video generation by integrating structured state-space models (SSMs) into diffusion models, reducing computational costs and improving performance over traditional attention-based methods.

Contribution

The paper demonstrates that bidirectional SSMs can effectively replace attention layers in diffusion models for video generation, enabling longer sequences with less memory and better quality.

Findings

SSM-based models require less memory for sequences up to 256 frames.

SSM models achieve comparable or better FVD scores than attention-based models.

Bidirectionality in SSMs enhances temporal feature capturing in videos.

Abstract

Given the remarkable achievements in image generation through diffusion models, the research community has shown increasing interest in extending these models to video generation. Recent diffusion models for video generation have predominantly utilized attention layers to extract temporal features. However, attention layers are limited by their computational costs, which increase quadratically with the sequence length. This limitation presents significant challenges when generating longer video sequences using diffusion models. To overcome this challenge, we propose leveraging state-space models (SSMs) as temporal feature extractors. SSMs (e.g., Mamba) have recently gained attention as promising alternatives due to their linear-time memory consumption relative to sequence length. In line with previous research suggesting that using bidirectional SSMs is effective for understanding spatial features in image generation, we found that bidirectionality is also beneficial for capturing temporal features in video data, rather than relying on traditional unidirectional SSMs. We conducted comprehensive evaluations on multiple long-term video datasets, such as MineRL Navigate, across various model sizes. For sequences up to 256 frames, SSM-based models require less memory to achieve the same FVD as attention-based models. Moreover, SSM-based models often deliver better performance with comparable GPU memory usage. Our codes are available at https://github.com/shim0114/SSM-Meets-Video-Diffusion-Models.