Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding

TL;DR

This paper introduces an object-driven one-shot fine-tuning method for text-to-image diffusion models that uses prototypical embeddings and class regularization to improve the generation of novel objects with high fidelity from a single image.

Contribution

It proposes a novel one-shot fine-tuning approach using prototypical embeddings and class regularization to enhance generalizability and fidelity in object-specific text-to-image generation.

Findings

Outperforms existing methods in object implantation quality

Effectively handles multiple objects in generated images

Maintains prior object class knowledge during fine-tuning

Abstract

As large-scale text-to-image generation models have made remarkable progress in the field of text-to-image generation, many fine-tuning methods have been proposed. However, these models often struggle with novel objects, especially with one-shot scenarios. Our proposed method aims to address the challenges of generalizability and fidelity in an object-driven way, using only a single input image and the object-specific regions of interest. To improve generalizability and mitigate overfitting, in our paradigm, a prototypical embedding is initialized based on the object's appearance and its class, before fine-tuning the diffusion model. And during fine-tuning, we propose a class-characterizing regularization to preserve prior knowledge of object classes. To further improve fidelity, we introduce object-specific loss, which can also use to implant multiple objects. Overall, our proposed object-driven method for implanting new objects can integrate seamlessly with existing concepts as well as with high fidelity and generalization. Our method outperforms several existing works. The code will be released.