Diffusion Model as Representation Learner

TL;DR

This paper explores the representation capabilities of Diffusion Probabilistic Models (DPMs) and introduces RepFusion, a novel knowledge transfer method that leverages DPMs for recognition tasks, outperforming existing methods.

Contribution

The paper provides an in-depth analysis of DPMs as autoencoders and proposes RepFusion, a new paradigm for transferring knowledge from DPMs to recognition models using reinforcement learning.

Findings

DPMs inherently function as denoising autoencoders.

RepFusion improves recognition performance across multiple benchmarks.

Knowledge transfer from DPMs enhances recognition tasks.

Abstract

Diffusion Probabilistic Models (DPMs) have recently demonstrated impressive results on various generative tasks.Despite its promises, the learned representations of pre-trained DPMs, however, have not been fully understood. In this paper, we conduct an in-depth investigation of the representation power of DPMs, and propose a novel knowledge transfer method that leverages the knowledge acquired by generative DPMs for recognition tasks. Our study begins by examining the feature space of DPMs, revealing that DPMs are inherently denoising autoencoders that balance the representation learning with regularizing model capacity. To this end, we introduce a novel knowledge transfer paradigm named RepFusion. Our paradigm extracts representations at different time steps from off-the-shelf DPMs and dynamically employs them as supervision for student networks, in which the optimal time is determined through reinforcement learning. We evaluate our approach on several image classification, semantic segmentation, and landmark detection benchmarks, and demonstrate that it outperforms state-of-the-art methods. Our results uncover the potential of DPMs as a powerful tool for representation learning and provide insights into the usefulness of generative models beyond sample generation. The code is available at \url{https://github.com/Adamdad/Repfusion}.