Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models

TL;DR

This paper introduces a fast, encoder-based framework for zero-shot image customization that generates high-quality, diverse images of user-specified objects without test-time optimization, leveraging object-specific embeddings.

Contribution

It presents a novel encoder-based approach with a regularized joint training scheme and caption generation to faithfully incorporate object identity into text-to-image models.

Findings

Produces high-quality, diverse images of customized objects

Eliminates the need for test-time optimization

Maintains object fidelity and appearance diversity

Abstract

This paper proposes a method for generating images of customized objects specified by users. The method is based on a general framework that bypasses the lengthy optimization required by previous approaches, which often employ a per-object optimization paradigm. Our framework adopts an encoder to capture high-level identifiable semantics of objects, producing an object-specific embedding with only a single feed-forward pass. The acquired object embedding is then passed to a text-to-image synthesis model for subsequent generation. To effectively blend a object-aware embedding space into a well developed text-to-image model under the same generation context, we investigate different network designs and training strategies, and propose a simple yet effective regularized joint training scheme with an object identity preservation loss. Additionally, we propose a caption generation scheme that become a critical piece in fostering object specific embedding faithfully reflected into the generation process, while keeping control and editing abilities. Once trained, the network is able to produce diverse content and styles, conditioned on both texts and objects. We demonstrate through experiments that our proposed method is able to synthesize images with compelling output quality, appearance diversity, and object fidelity, without the need of test-time optimization. Systematic studies are also conducted to analyze our models, providing insights for future work.