Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation

TL;DR

Control-CLIP introduces a decoupled fine-tuning framework for CLIP that separately learns category and style semantics, enhancing domain-specific image generation without altering the original diffusion model.

Contribution

It proposes a novel decoupled CLIP fine-tuning method with a modified cross-attention mechanism for precise domain guidance in text-to-image diffusion models.

Findings

Effective in generating domain-specific images with various styles

Preserves original diffusion model performance and diversity

Robust plug-and-play capability across multiple domains

Abstract

Text-to-image diffusion models have shown remarkable capabilities of generating high-quality images closely aligned with textual inputs. However, the effectiveness of text guidance heavily relies on the CLIP text encoder, which is trained to pay more attention to general content but struggles to capture semantics in specific domains like styles. As a result, generation models tend to fail on prompts like "a photo of a cat in Pokemon style" in terms of simply producing images depicting "a photo of a cat". To fill this gap, we propose Control-CLIP, a novel decoupled CLIP fine-tuning framework that enables the CLIP model to learn the meaning of category and style in a complement manner. With specially designed fine-tuning tasks on minimal data and a modified cross-attention mechanism, Control-CLIP can precisely guide the diffusion model to a specific domain. Moreover, the parameters of the diffusion model remain unchanged at all, preserving the original generation performance and diversity. Experiments across multiple domains confirm the effectiveness of our approach, particularly highlighting its robust plug-and-play capability in generating content with various specific styles.