PatchScaler: An Efficient Patch-Independent Diffusion Model for Image Super-Resolution

TL;DR

PatchScaler introduces a patch-adaptive diffusion approach for image super-resolution that reduces inference time by customizing sampling steps based on patch difficulty, enhancing efficiency and quality.

Contribution

It proposes a novel patch-independent diffusion pipeline with adaptive sampling and texture prompts, improving super-resolution efficiency and quality over existing methods.

Findings

Achieves faster inference with comparable or better quality.

Effectively groups patches by reconstruction difficulty.

Utilizes texture prompts for improved patch reconstruction.

Abstract

While diffusion models significantly improve the perceptual quality of super-resolved images, they usually require a large number of sampling steps, resulting in high computational costs and long inference times. Recent efforts have explored reasonable acceleration schemes by reducing the number of sampling steps. However, these approaches treat all regions of the image equally, overlooking the fact that regions with varying levels of reconstruction difficulty require different sampling steps. To address this limitation, we propose PatchScaler, an efficient patch-independent diffusion pipeline for single image super-resolution. Specifically, PatchScaler introduces a Patch-adaptive Group Sampling (PGS) strategy that groups feature patches by quantifying their reconstruction difficulty and establishes shortcut paths with different sampling configurations for each group. To further optimize the patch-level reconstruction process of PGS, we propose a texture prompt that provides rich texture conditional information to the diffusion model. The texture prompt adaptively retrieves texture priors for the target patch from a common reference texture memory. Extensive experiments show that our PatchScaler achieves superior performance in both quantitative and qualitative evaluations, while significantly speeding up inference. Our code will be available at \url{https://github.com/yongliuy/PatchScaler}.