CusConcept: Customized Visual Concept Decomposition with Diffusion Models

TL;DR

CusConcept introduces a two-stage diffusion-based framework for decomposing visual concepts from a single image, enabling customized, multi-perspective image generation and concept understanding.

Contribution

It proposes a novel two-stage method for visual concept decomposition using diffusion models, including a vocabulary-guided mechanism and joint refinement, along with a new benchmark for evaluation.

Findings

Effective high-quality image generation of decomposed concepts

Accurate lexical predictions for concepts

Superior performance demonstrated through extensive experiments

Abstract

Enabling generative models to decompose visual concepts from a single image is a complex and challenging problem. In this paper, we study a new and challenging task, customized concept decomposition, wherein the objective is to leverage diffusion models to decompose a single image and generate visual concepts from various perspectives. To address this challenge, we propose a two-stage framework, CusConcept (short for Customized Visual Concept Decomposition), to extract customized visual concept embedding vectors that can be embedded into prompts for text-to-image generation. In the first stage, CusConcept employs a vocabulary-guided concept decomposition mechanism to build vocabularies along human-specified conceptual axes. The decomposed concepts are obtained by retrieving corresponding vocabularies and learning anchor weights. In the second stage, joint concept refinement is performed to enhance the fidelity and quality of generated images. We further curate an evaluation benchmark for assessing the performance of the open-world concept decomposition task. Our approach can effectively generate high-quality images of the decomposed concepts and produce related lexical predictions as secondary outcomes. Extensive qualitative and quantitative experiments demonstrate the effectiveness of CusConcept.