DreamArtist++: Controllable One-Shot Text-to-Image Generation via Positive-Negative Adapter

TL;DR

DreamArtist++ introduces a positive-negative prompt-tuning framework for one-shot text-to-image generation, effectively balancing controllability, fidelity, and diversity using a joint optimization of positive and negative adapters on pre-trained diffusion models.

Contribution

The paper presents a novel positive-negative prompt-tuning strategy that improves one-shot image generation quality and controllability, outperforming existing methods.

Findings

Achieved superior image fidelity and diversity compared to existing methods.

Effectively captures reference characteristics with only one example.

Demonstrated versatility in concept composition and image editing tasks.

Abstract

State-of-the-arts text-to-image generation models such as Imagen and Stable Diffusion Model have succeed remarkable progresses in synthesizing high-quality, feature-rich images with high resolution guided by human text prompts. Since certain characteristics of image content \emph{e.g.}, very specific object entities or styles, are very hard to be accurately described by text, some example-based image generation approaches have been proposed, \emph{i.e.} generating new concepts based on absorbing the salient features of a few input references. Despite of acknowledged successes, these methods have struggled on accurately capturing the reference examples' characteristics while keeping diverse and high-quality image generation, particularly in the one-shot scenario (\emph{i.e.} given only one reference). To tackle this problem, we propose a simple yet effective framework, namely DreamArtist, which adopts a novel positive-negative prompt-tuning learning strategy on the pre-trained diffusion model, and it has shown to well handle the trade-off between the accurate controllability and fidelity of image generation with only one reference example. Specifically, our proposed framework incorporates both positive and negative embeddings or adapters and optimizes them in a joint manner. The positive part aggressively captures the salient characteristics of the reference image to drive diversified generation and the negative part rectifies inadequacies from the positive part. We have conducted extensive experiments and evaluated the proposed method from image similarity (fidelity) and diversity, generation controllability, and style cloning. And our DreamArtist has achieved a superior generation performance over existing methods. Besides, our additional evaluation on extended tasks, including concept compositions and prompt-guided image editing, demonstrates its effectiveness for more applications.