AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration

TL;DR

AsymRnR is a training-free, model-agnostic method that accelerates video diffusion transformers by asymmetrically reducing feature token redundancies, achieving significant speedups with minimal quality loss.

Contribution

It introduces a novel asymmetrical token reduction technique for video DiTs that does not require fine-tuning and adapts reduction schedules for efficient acceleration.

Findings

Achieves substantial speedup in video DiTs with negligible quality degradation.

Demonstrates effectiveness across multiple state-of-the-art video DiT models.

Provides theoretical analysis and extensive experiments validating the method.

Abstract

Diffusion Transformers (DiTs) have proven effective in generating high-quality videos but are hindered by high computational costs. Existing video DiT sampling acceleration methods often rely on costly fine-tuning or exhibit limited generalization capabilities. We propose Asymmetric Reduction and Restoration (AsymRnR), a training-free and model-agnostic method to accelerate video DiTs. It builds on the observation that redundancies of feature tokens in DiTs vary significantly across different model blocks, denoising steps, and feature types. Our AsymRnR asymmetrically reduces redundant tokens in the attention operation, achieving acceleration with negligible degradation in output quality and, in some cases, even improving it. We also tailored a reduction schedule to distribute the reduction across components adaptively. To further accelerate this process, we introduce a matching cache for more efficient reduction. Backed by theoretical foundations and extensive experimental validation, AsymRnR integrates into state-of-the-art video DiTs and offers substantial speedup.