HoliSDiP: Image Super-Resolution via Holistic Semantics and Diffusion Prior

TL;DR

HoliSDiP enhances real-world image super-resolution by integrating semantic segmentation and diffusion models, providing precise spatial and textual guidance to improve image quality and reduce noise in outputs.

Contribution

The paper introduces a novel framework combining semantic segmentation and diffusion priors for improved real-world image super-resolution.

Findings

Significant image quality improvements across various scenarios.

Reduced noise and better spatial control in super-resolved images.

Effective use of semantic labels and segmentation masks for guidance.

Abstract

Text-to-image diffusion models have emerged as powerful priors for real-world image super-resolution (Real-ISR). However, existing methods may produce unintended results due to noisy text prompts and their lack of spatial information. In this paper, we present HoliSDiP, a framework that leverages semantic segmentation to provide both precise textual and spatial guidance for diffusion-based Real-ISR. Our method employs semantic labels as concise text prompts while introducing dense semantic guidance through segmentation masks and our proposed Segmentation-CLIP Map. Extensive experiments demonstrate that HoliSDiP achieves significant improvement in image quality across various Real-ISR scenarios through reduced prompt noise and enhanced spatial control.