Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion

TL;DR

This paper introduces a motion-adaptive temporal attention mechanism for lightweight video generation using frozen Stable Diffusion models, dynamically adjusting attention based on motion to improve temporal consistency and detail.

Contribution

It proposes a novel, parameter-efficient temporal attention method that adapts to motion content, enhancing video quality without extensive additional training.

Findings

Achieves competitive results with only 25.8M additional parameters.

Implicit temporal regularization from the denoising objective outperforms explicit losses.

Provides a practical control for diverse generation behaviors via noise correlation and motion amplitude.

Abstract

We present a motion-adaptive temporal attention mechanism for parameter-efficient video generation built upon frozen Stable Diffusion models. Rather than treating all video content uniformly, our method dynamically adjusts temporal attention receptive fields based on estimated motion content: high-motion sequences attend locally across frames to preserve rapidly changing details, while low-motion sequences attend globally to enforce scene consistency. We inject lightweight temporal attention modules into all UNet transformer blocks via a cascaded strategy -- global attention in down-sampling and middle blocks for semantic stabilization, motion-adaptive attention in up-sampling blocks for fine-grained refinement. Combined with temporally correlated noise initialization and motion-aware gating, the system adds only 25.8M trainable parameters (2.9\% of the base UNet) while achieving competitive results on WebVid validation when trained on 100K videos. We demonstrate that the standard denoising objective alone provides sufficient implicit temporal regularization, outperforming approaches that add explicit temporal consistency losses. Our ablation studies reveal a clear trade-off between noise correlation and motion amplitude, providing a practical inference-time control for diverse generation behaviors.