HybridBooth: Hybrid Prompt Inversion for Efficient Subject-Driven Generation

TL;DR

HybridBooth introduces a hybrid prompt inversion framework that efficiently personalizes text-to-image generation by combining optimization and regression techniques, enabling fast and effective subject-driven image synthesis from minimal input.

Contribution

The paper proposes HybridBooth, a novel two-stage hybrid approach that improves the speed and quality of subject-driven generation in diffusion models.

Findings

Enables effective inversion from a single image.

Maintains generalization capabilities of the model.

Provides faster and more accurate personalization.

Abstract

Recent advancements in text-to-image diffusion models have shown remarkable creative capabilities with textual prompts, but generating personalized instances based on specific subjects, known as subject-driven generation, remains challenging. To tackle this issue, we present a new hybrid framework called HybridBooth, which merges the benefits of optimization-based and direct-regression methods. HybridBooth operates in two stages: the Word Embedding Probe, which generates a robust initial word embedding using a fine-tuned encoder, and the Word Embedding Refinement, which further adapts the encoder to specific subject images by optimizing key parameters. This approach allows for effective and fast inversion of visual concepts into textual embedding, even from a single image, while maintaining the model's generalization capabilities.