Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation

TL;DR

This paper introduces a training-free latent inter-frame pruning method with attention recovery to improve the efficiency of diffusion-based video generation, reducing computation while maintaining quality.

Contribution

It proposes a novel pruning framework that skips redundant latent computations and an attention recovery mechanism to mitigate artifacts, bridging traditional video compression and modern diffusion models.

Findings

Increases video editing throughput by 1.44×

Achieves 12.44 FPS on NVIDIA RTX 6000

Maintains video quality despite pruning

Abstract

Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications. In this paper, we observe that video latents encoded via an autoencoder under the Latent Diffusion Model (LDM) framework contain redundancy along the temporal axis. Analogous to how traditional video compression algorithms avoid transmitting redundant frame data, we propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput. However, direct pruning results in visual artifacts due to the discrepancy between full-sequence training and pruned inference. To resolve these artifacts, we propose an Attention Recovery mechanism to bridge the train-inference gap. With our proposed method, we increase video editing throughput by 1.44$\times$, achieving 12.44 FPS on an NVIDIA RTX 6000 while maintaining video quality. We hope our work inspires further research into integrating traditional video compression methods with modern video generation pipelines. This work is a preliminary work on Training-free Latent Inter-Frame Pruning with Attention Recovery.