Fine Tuning Text-to-Image Diffusion Models for Correcting Anomalous Images

TL;DR

This paper presents a fine-tuning approach for Stable Diffusion 3 to reduce aberrant images in text-to-image generation, improving visual quality and user preference.

Contribution

It introduces a fine-tuning method using DreamBooth to enhance the reliability of text-to-image models for specific prompts.

Findings

Improved SSIM, PSNR, and FID scores with fine-tuning.

Higher user preference for fine-tuned images.

Better visual quality in generated images.

Abstract

Since the advent of GANs and VAEs, image generation models have continuously evolved, opening up various real-world applications with the introduction of Stable Diffusion and DALL-E models. These text-to-image models can generate high-quality images for fields such as art, design, and advertising. However, they often produce aberrant images for certain prompts. This study proposes a method to mitigate such issues by fine-tuning the Stable Diffusion 3 model using the DreamBooth technique. Experimental results targeting the prompt "lying on the grass/street" demonstrate that the fine-tuned model shows improved performance in visual evaluation and metrics such as Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Frechet Inception Distance (FID). User surveys also indicated a higher preference for the fine-tuned model. This research is expected to make contributions to enhancing the practicality and reliability of text-to-image models.