Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study

TL;DR

This paper systematically investigates redundancy in Diffusion Transformers, revealing significant model-specific variations and stability within models, and introduces a tool for analyzing redundancy to aid in developing tailored caching strategies.

Contribution

It provides a comprehensive analysis of redundancy patterns across various DiT models and offers a new tool for model-specific redundancy analysis.

Findings

Redundancy varies significantly across different DiT models.

Within a single model, redundancy distribution remains stable despite input or scheduling changes.

A new tool is introduced for analyzing model-specific redundancy patterns.

Abstract

The increased model capacity of Diffusion Transformers (DiTs) and the demand for generating higher resolutions of images and videos have led to a significant rise in inference latency, impacting real-time performance adversely. While prior research has highlighted the presence of high similarity in activation values between adjacent diffusion steps (referred to as redundancy) and proposed various caching mechanisms to mitigate computational overhead, the exploration of redundancy in existing literature remains limited, with findings often not generalizable across different DiT models. This study aims to address this gap by conducting a comprehensive investigation into redundancy across a broad spectrum of mainstream DiT models. Our experimental analysis reveals substantial variations in the distribution of redundancy across diffusion steps among different DiT models. Interestingly, within a single model, the redundancy distribution remains stable regardless of variations in input prompts, step counts, or scheduling strategies. Given the lack of a consistent pattern across diverse models, caching strategies designed for a specific group of models may not easily transfer to others. To overcome this challenge, we introduce a tool for analyzing the redundancy of individual models, enabling subsequent research to develop tailored caching strategies for specific model architectures. The project is publicly available at https://github.com/xdit-project/DiTCacheAnalysis.