HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models

TL;DR

HiFi Tuner introduces a novel, efficient fine-tuning method for diffusion models that significantly improves personalized image generation fidelity and enables subject substitution through text, outperforming previous approaches.

Contribution

The paper presents HiFi Tuner, a parameter-efficient fine-tuning framework with novel techniques like mask guidance and reference-guided generation to enhance subject fidelity in personalized diffusion-based image synthesis.

Findings

Improves CLIP-T score by 3.6 points over Textual Inversion

Enhances DINO score by 9.6 points over Textual Inversion

Sets new state-of-the-art results on DreamBooth dataset

Abstract

This paper explores advancements in high-fidelity personalized image generation through the utilization of pre-trained text-to-image diffusion models. While previous approaches have made significant strides in generating versatile scenes based on text descriptions and a few input images, challenges persist in maintaining the subject fidelity within the generated images. In this work, we introduce an innovative algorithm named HiFi Tuner to enhance the appearance preservation of objects during personalized image generation. Our proposed method employs a parameter-efficient fine-tuning framework, comprising a denoising process and a pivotal inversion process. Key enhancements include the utilization of mask guidance, a novel parameter regularization technique, and the incorporation of step-wise subject representations to elevate the sample fidelity. Additionally, we propose a reference-guided generation approach that leverages the pivotal inversion of a reference image to mitigate unwanted subject variations and artifacts. We further extend our method to a novel image editing task: substituting the subject in an image through textual manipulations. Experimental evaluations conducted on the DreamBooth dataset using the Stable Diffusion model showcase promising results. Fine-tuning solely on textual embeddings improves CLIP-T score by 3.6 points and improves DINO score by 9.6 points over Textual Inversion. When fine-tuning all parameters, HiFi Tuner improves CLIP-T score by 1.2 points and improves DINO score by 1.2 points over DreamBooth, establishing a new state of the art.