OmniPrism: Learning Disentangled Visual Concept for Image Generation

TL;DR

OmniPrism introduces a novel approach for disentangling visual concepts guided by natural language, enabling more creative and accurate image generation with high fidelity to prompts.

Contribution

The paper presents a new contrastive orthogonal disentangled training pipeline and a paired dataset for learning disentangled concepts in diffusion models.

Findings

Achieves high-quality, concept-disentangled image generation.

Effectively incorporates multiple concepts guided by natural language.

Demonstrates superior performance over existing methods.

Abstract

Creative visual concept generation often draws inspiration from specific concepts in a reference image to produce relevant outcomes. However, existing methods are typically constrained to single-aspect concept generation or are easily disrupted by irrelevant concepts in multi-aspect concept scenarios, leading to concept confusion and hindering creative generation. To address this, we propose OmniPrism, a visual concept disentangling approach for creative image generation. Our method learns disentangled concept representations guided by natural language and trains a diffusion model to incorporate these concepts. We utilize the rich semantic space of a multimodal extractor to achieve concept disentanglement from given images and concept guidance. To disentangle concepts with different semantics, we construct a paired concept disentangled dataset (PCD-200K), where each pair shares the same concept such as content, style, and composition. We learn disentangled concept representations through our contrastive orthogonal disentangled (COD) training pipeline, which are then injected into additional diffusion cross-attention layers for generation. A set of block embeddings is designed to adapt each block's concept domain in the diffusion models. Extensive experiments demonstrate that our method can generate high-quality, concept-disentangled results with high fidelity to text prompts and desired concepts.