Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Model

TL;DR

This paper introduces cached adaptive token merging (CA-ToMe), a method that reduces computational costs in diffusion models by adaptively merging similar tokens with caching, achieving faster inference without sacrificing quality.

Contribution

The paper proposes a novel adaptive token merging technique with caching for diffusion models, improving speed while maintaining image quality, and is training-free.

Findings

Achieves 1.24x speedup in denoising process

Maintains the same FID scores as existing methods

Operates without additional training

Abstract

Diffusion models have emerged as a promising approach for generating high-quality, high-dimensional images. Nevertheless, these models are hindered by their high computational cost and slow inference, partly due to the quadratic computational complexity of the self-attention mechanisms with respect to input size. Various approaches have been proposed to address this drawback. One such approach focuses on reducing the number of tokens fed into the self-attention, known as token merging (ToMe). In our method, which is called cached adaptive token merging(CA-ToMe), we calculate the similarity between tokens and then merge the r proportion of the most similar tokens. However, due to the repetitive patterns observed in adjacent steps and the variation in the frequency of similarities, we aim to enhance this approach by implementing an adaptive threshold for merging tokens and adding a caching mechanism that stores similar pairs across several adjacent steps. Empirical results demonstrate that our method operates as a training-free acceleration method, achieving a speedup factor of 1.24 in the denoising process while maintaining the same FID scores compared to existing approaches.