DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery

TL;DR

This paper introduces DiffusionSeg, a novel framework that leverages pre-trained diffusion models for unsupervised object discovery tasks like saliency segmentation and object localization, overcoming structural and data limitations.

Contribution

It proposes a two-stage synthesis-exploitation approach with a training-free mask generation method and an inversion technique to adapt diffusion models for discriminative tasks.

Findings

DiffusionSeg outperforms existing methods in unsupervised object discovery.

The synthesis stage effectively generates training data to improve performance.

The inversion technique bridges the gap between generative and discriminative models.

Abstract

Learning from a large corpus of data, pre-trained models have achieved impressive progress nowadays. As popular generative pre-training, diffusion models capture both low-level visual knowledge and high-level semantic relations. In this paper, we propose to exploit such knowledgeable diffusion models for mainstream discriminative tasks, i.e., unsupervised object discovery: saliency segmentation and object localization. However, the challenges exist as there is one structural difference between generative and discriminative models, which limits the direct use. Besides, the lack of explicitly labeled data significantly limits performance in unsupervised settings. To tackle these issues, we introduce DiffusionSeg, one novel synthesis-exploitation framework containing two-stage strategies. To alleviate data insufficiency, we synthesize abundant images, and propose a novel training-free AttentionCut to obtain masks in the first synthesis stage. In the second exploitation stage, to bridge the structural gap, we use the inversion technique, to map the given image back to diffusion features. These features can be directly used by downstream architectures. Extensive experiments and ablation studies demonstrate the superiority of adapting diffusion for unsupervised object discovery.