TIMERIPPLE: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

TL;DR

This paper introduces TIMERIPPLE, a method that accelerates video diffusion transformers by exploiting spatio-temporal correlations in latent space, achieving 85% computational savings with minimal quality loss.

Contribution

It presents a novel adaptive reuse strategy that leverages inherent spatio-temporal correlations to efficiently approximate self-attention in vDiTs.

Findings

Achieves 85% reduction in inference computation.

Maintains video quality with less than 0.06% loss on VBench.

Applicable across multiple vDiT architectures.

Abstract

The recent surge in video generation has shown the growing demand for high-quality video synthesis using large vision models. Existing video generation models are predominantly based on the video diffusion transformer (vDiT), however, they suffer from substantial inference delay due to self-attention. While prior studies have focused on reducing redundant computations in self-attention, they often overlook the inherent spatio-temporal correlations in video streams and directly leverage sparsity patterns from large language models to reduce attention computations. In this work, we take a principled approach to accelerate self-attention in vDiTs by leveraging the spatio-temporal correlations in the latent space. We show that the attention patterns within vDiT are primarily due to the dominant spatial and temporal correlations at the token channel level. Based on this insight, we propose a lightweight and adaptive reuse strategy that approximates attention computations by reusing partial attention scores of spatially or temporally correlated tokens along individual channels. We demonstrate that our method achieves significantly higher computational savings (85\%) compared to state-of-the-art techniques over 4 vDiTs, while preserving almost identical video quality ($<$0.06\% loss on VBench).