Prompt-tuning latent diffusion models for inverse problems

TL;DR

This paper introduces P2L, a prompt tuning method for latent diffusion models that enhances inverse problem solving by optimizing text prompts and maintaining latent variables within the encoder's range, leading to improved image reconstruction.

Contribution

The paper presents a novel prompt tuning approach combined with latent space projection to improve inverse problem solutions using latent diffusion models.

Findings

P2L outperforms existing inverse problem solvers on multiple tasks.

Prompt tuning improves the faithfulness of generated images.

Latent space projection reduces artifacts in reconstructed images.

Abstract

We propose a new method for solving imaging inverse problems using text-to-image latent diffusion models as general priors. Existing methods using latent diffusion models for inverse problems typically rely on simple null text prompts, which can lead to suboptimal performance. To address this limitation, we introduce a method for prompt tuning, which jointly optimizes the text embedding on-the-fly while running the reverse diffusion process. This allows us to generate images that are more faithful to the diffusion prior. In addition, we propose a method to keep the evolution of latent variables within the range space of the encoder, by projection. This helps to reduce image artifacts, a major problem when using latent diffusion models instead of pixel-based diffusion models. Our combined method, called P2L, outperforms both image- and latent-diffusion model-based inverse problem solvers on a variety of tasks, such as super-resolution, deblurring, and inpainting.