MSC: Multi-Scale Spatio-Temporal Causal Attention for Autoregressive Video Diffusion

TL;DR

This paper introduces MSC, a multi-scale causal attention framework for autoregressive video diffusion that reduces computational complexity and enables efficient high-resolution video generation with rich semantics.

Contribution

It proposes a novel multi-scale causal attention mechanism that improves efficiency and supports autoregressive long video generation without violating temporal order.

Findings

Reduces computational complexity in video diffusion models

Enables high-resolution video generation with rich semantics

Supports autoregressive long video synthesis

Abstract

Diffusion transformers enable flexible generative modeling for video. However, it is still technically challenging and computationally expensive to generate high-resolution videos with rich semantics and complex motion. Similar to languages, video data are also auto-regressive by nature, so it is counter-intuitive to use attention mechanism with bi-directional dependency in the model. Here we propose a Multi-Scale Causal (MSC) framework to address these problems. Specifically, we introduce multiple resolutions in the spatial dimension and high-low frequencies in the temporal dimension to realize efficient attention calculation. Furthermore, attention blocks on multiple scales are combined in a controlled way to allow causal conditioning on noisy image frames for diffusion training, based on the idea that noise destroys information at different rates on different resolutions. We theoretically show that our approach can greatly reduce the computational complexity and enhance the efficiency of training. The causal attention diffusion framework can also be used for auto-regressive long video generation, without violating the natural order of frame sequences.