ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction

TL;DR

ConceptExpress introduces an unsupervised method to extract and recreate multiple concepts from a single image using pretrained diffusion models, eliminating the need for human annotations.

Contribution

It proposes a novel unsupervised concept extraction framework leveraging diffusion models' inherent capabilities, including spatial localization and token association, for multi-concept understanding.

Findings

Effective localization of salient concepts via diffusion self-attention

Discriminative token learning for individual concepts

Promising results on the UCE evaluation protocol

Abstract

While personalized text-to-image generation has enabled the learning of a single concept from multiple images, a more practical yet challenging scenario involves learning multiple concepts within a single image. However, existing works tackling this scenario heavily rely on extensive human annotations. In this paper, we introduce a novel task named Unsupervised Concept Extraction (UCE) that considers an unsupervised setting without any human knowledge of the concepts. Given an image that contains multiple concepts, the task aims to extract and recreate individual concepts solely relying on the existing knowledge from pretrained diffusion models. To achieve this, we present ConceptExpress that tackles UCE by unleashing the inherent capabilities of pretrained diffusion models in two aspects. Specifically, a concept localization approach automatically locates and disentangles salient concepts by leveraging spatial correspondence from diffusion self-attention; and based on the lookup association between a concept and a conceptual token, a concept-wise optimization process learns discriminative tokens that represent each individual concept. Finally, we establish an evaluation protocol tailored for the UCE task. Extensive experiments demonstrate that ConceptExpress is a promising solution to the UCE task. Our code and data are available at: https://github.com/haoosz/ConceptExpress