Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration

TL;DR

Diff-Restorer introduces a universal diffusion-based image restoration method that leverages visual prompts and prior knowledge to effectively recover high-quality images from various degraded inputs, outperforming existing techniques.

Contribution

The paper presents a novel diffusion model framework utilizing visual prompts and prior knowledge for universal image restoration, addressing limitations of over-smoothing and lack of realism.

Findings

Effective restoration across multiple degradation types

Outperforms existing methods in qualitative and quantitative metrics

Generates high perceptual quality restored images

Abstract

Image restoration is a classic low-level problem aimed at recovering high-quality images from low-quality images with various degradations such as blur, noise, rain, haze, etc. However, due to the inherent complexity and non-uniqueness of degradation in real-world images, it is challenging for a model trained for single tasks to handle real-world restoration problems effectively. Moreover, existing methods often suffer from over-smoothing and lack of realism in the restored results. To address these issues, we propose Diff-Restorer, a universal image restoration method based on the diffusion model, aiming to leverage the prior knowledge of Stable Diffusion to remove degradation while generating high perceptual quality restoration results. Specifically, we utilize the pre-trained visual language model to extract visual prompts from degraded images, including semantic and degradation embeddings. The semantic embeddings serve as content prompts to guide the diffusion model for generation. In contrast, the degradation embeddings modulate the Image-guided Control Module to generate spatial priors for controlling the spatial structure of the diffusion process, ensuring faithfulness to the original image. Additionally, we design a Degradation-aware Decoder to perform structural correction and convert the latent code to the pixel domain. We conducted comprehensive qualitative and quantitative analysis on restoration tasks with different degradations, demonstrating the effectiveness and superiority of our approach.