Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations

TL;DR

This paper investigates the limitations of Diffusion Transformers in visual correspondence due to massive activation concentration and introduces DiTF, a normalization framework that improves their performance significantly.

Contribution

The paper uncovers the link between massive activations and AdaLN in DiTs and proposes DiTF, a training-free method that enhances feature extraction for better correspondence results.

Findings

DiTF outperforms DINO and SD-based models in visual correspondence tasks.

Massive activations are linked to AdaLN in DiTs and can be mitigated.

State-of-the-art results achieved on Spair-71k and AP-10K-C.S.

Abstract

Pre-trained stable diffusion models (SD) have shown great advances in visual correspondence. In this paper, we investigate the capabilities of Diffusion Transformers (DiTs) for accurate dense correspondence. Distinct from SD, DiTs exhibit a critical phenomenon in which very few feature activations exhibit significantly larger values than others, known as \textit{massive activations}, leading to uninformative representations and significant performance degradation for DiTs. The massive activations consistently concentrate at very few fixed dimensions across all image patch tokens, holding little local information. We analyze these dimension-concentrated massive activations and uncover that their concentration is inherently linked to the Adaptive Layer Normalization (AdaLN) in DiTs. Building on these findings, we propose the \textbf{Di}ffusion \textbf{T}ransformer \textbf{F}eature (DiTF), a training-free AdaLN-based framework that extracts semantically discriminative features from DiTs. Specifically, DiTF leverages AdaLN to adaptively localize and normalize massive activations through channel-wise modulation. Furthermore, a channel discard strategy is introduced to mitigate the adverse effects of massive activations. Experimental results demonstrate that our DiTF outperforms both DINO and SD-based models and establishes a new state-of-the-art performance for DiTs in different visual correspondence tasks (\eg, with +9.4\% on Spair-71k and +4.4\% on AP-10K-C.S.).