SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic Image Restoration

TL;DR

SSP-IR introduces a novel diffusion-based image restoration method that leverages semantic and structure priors from low-quality images, utilizing multimodal language models and FFT constraints to enhance perceptual quality, semantic fidelity, and structural accuracy.

Contribution

The paper proposes a new approach that fully exploits semantic and structure priors from degraded images using multimodal models and FFT constraints, improving restoration quality.

Findings

Outperforms state-of-the-art methods on synthetic datasets

Achieves superior perceptual and structural restoration results

Effective in real-world image restoration scenarios

Abstract

Realistic image restoration is a crucial task in computer vision, and diffusion-based models for image restoration have garnered significant attention due to their ability to produce realistic results. Restoration can be seen as a controllable generation conditioning on priors. However, due to the severity of image degradation, existing diffusion-based restoration methods cannot fully exploit priors from low-quality images and still have many challenges in perceptual quality, semantic fidelity, and structure accuracy. Based on the challenges, we introduce a novel image restoration method, SSP-IR. Our approach aims to fully exploit semantic and structure priors from low-quality images to guide the diffusion model in generating semantically faithful and structurally accurate natural restoration results. Specifically, we integrate the visual comprehension capabilities of Multimodal Large Language Models (explicit) and the visual representations of the original image (implicit) to acquire accurate semantic prior. To extract degradation-independent structure prior, we introduce a Processor with RGB and FFT constraints to extract structure prior from the low-quality images, guiding the diffusion model and preventing the generation of unreasonable artifacts. Lastly, we employ a multi-level attention mechanism to integrate the acquired semantic and structure priors. The qualitative and quantitative results demonstrate that our method outperforms other state-of-the-art methods overall on both synthetic and real-world datasets. Our project page is https://zyhrainbow.github.io/projects/SSP-IR.