Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features

TL;DR

This paper evaluates a broader range of diffusion model activations for discriminative tasks, introduces properties to filter ineffective features, and proposes feature selection methods that outperform state-of-the-art approaches.

Contribution

It extends activation evaluation to include new types like queries and keys, and introduces properties for qualitative filtering, advancing feature selection in diffusion models.

Findings

Identified three universal properties of diffusion activations.

Proposed effective feature selection methods outperform SOTA.

Validated methods across multiple discriminative tasks.

Abstract

Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at https://github.com/Darkbblue/generic-diffusion-feature.