PRedItOR: Text Guided Image Editing with Diffusion Prior

TL;DR

PRedItOR introduces a novel text-guided image editing method using a hybrid diffusion model that avoids fine-tuning or optimization, achieving high-quality, structure-preserving edits efficiently.

Contribution

It presents a diffusion prior model for conceptual text-guided image editing without fine-tuning, enhancing efficiency and flexibility over existing methods.

Findings

Achieves comparable or better results than baselines.

Does not require fine-tuning or optimization.

Enables structure-preserving edits using diffusion prior.

Abstract

Diffusion models have shown remarkable capabilities in generating high quality and creative images conditioned on text. An interesting application of such models is structure preserving text guided image editing. Existing approaches rely on text conditioned diffusion models such as Stable Diffusion or Imagen and require compute intensive optimization of text embeddings or fine-tuning the model weights for text guided image editing. We explore text guided image editing with a Hybrid Diffusion Model (HDM) architecture similar to DALLE-2. Our architecture consists of a diffusion prior model that generates CLIP image embedding conditioned on a text prompt and a custom Latent Diffusion Model trained to generate images conditioned on CLIP image embedding. We discover that the diffusion prior model can be used to perform text guided conceptual edits on the CLIP image embedding space without any finetuning or optimization. We combine this with structure preserving edits on the image decoder using existing approaches such as reverse DDIM to perform text guided image editing. Our approach, PRedItOR does not require additional inputs, fine-tuning, optimization or objectives and shows on par or better results than baselines qualitatively and quantitatively. We provide further analysis and understanding of the diffusion prior model and believe this opens up new possibilities in diffusion models research.