Improving Consistency in Diffusion Models for Image Super-Resolution

TL;DR

This paper introduces ConsisSR, a novel diffusion-based super-resolution method that improves semantic and training-inference consistency using a hybrid prompt adapter and time-aware latent augmentation, achieving state-of-the-art results.

Contribution

The paper proposes a new framework, ConsisSR, with a hybrid prompt adapter and time-aware latent augmentation to address semantic and training-inference inconsistencies in diffusion-based image super-resolution.

Findings

Achieves state-of-the-art performance among diffusion models.

Effectively reduces semantic inconsistency with CLIP image embeddings.

Bridges training-inference gap with TALA, improving reconstruction quality.

Abstract

Recent methods exploit the powerful text-to-image (T2I) diffusion models for real-world image super-resolution (Real-ISR) and achieve impressive results compared to previous models. However, we observe two kinds of inconsistencies in diffusion-based methods which hinder existing models from fully exploiting diffusion priors. The first is the semantic inconsistency arising from diffusion guidance. T2I generation focuses on semantic-level consistency with text prompts, while Real-ISR emphasizes pixel-level reconstruction from low-quality (LQ) images, necessitating more detailed semantic guidance from LQ inputs. The second is the training-inference inconsistency stemming from the DDPM, which improperly assumes high-quality (HQ) latent corrupted by Gaussian noise as denoising inputs for each timestep. To address these issues, we introduce ConsisSR to handle both semantic and training-inference consistencies. On the one hand, to address the semantic inconsistency, we proposed a Hybrid Prompt Adapter (HPA). Instead of text prompts with coarse-grained classification information, we leverage the more powerful CLIP image embeddings to explore additional color and texture guidance. On the other hand, we introduce Time-Aware Latent Augmentation (TALA) to bridge the training-inference inconsistency. Based on the probability function p(t), we accordingly enhance the SDSR training strategy. With LQ latent with Gaussian noise as inputs, our TALA not only focuses on diffusion noise but also refine the LQ latent towards the HQ counterpart. Our method demonstrates state-of-the-art performance among existing diffusion models. The code will be made publicly available.