Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images

TL;DR

Diffusion Brush is a novel editing tool based on latent diffusion models that allows targeted, efficient fine-tuning of specific regions in AI-generated images, improving editing precision and preserving image context.

Contribution

The paper introduces Diffusion Brush, a new method for region-specific image editing using latent diffusion models, addressing limitations of existing fine-tuning techniques.

Findings

Effective regional editing with minimal artifacts

Outperforms existing inpainting and editing tools in user studies

Preserves original image context during targeted modifications

Abstract

Text-to-image generative models have made remarkable advancements in generating high-quality images. However, generated images often contain undesirable artifacts or other errors due to model limitations. Existing techniques to fine-tune generated images are time-consuming (manual editing), produce poorly-integrated results (inpainting), or result in unexpected changes across the entire image (variation selection and prompt fine-tuning). In this work, we present Diffusion Brush, a Latent Diffusion Model-based (LDM) tool to efficiently fine-tune desired regions within an AI-synthesized image. Our method introduces new random noise patterns at targeted regions during the reverse diffusion process, enabling the model to efficiently make changes to the specified regions while preserving the original context for the rest of the image. We evaluate our method's usability and effectiveness through a user study with artists, comparing our technique against other state-of-the-art image inpainting techniques and editing software for fine-tuning AI-generated imagery.