Accelerating Diffusion-based Super-Resolution with Dynamic Time-Spatial Sampling

TL;DR

This paper introduces a novel time-spatial-aware sampling method that accelerates diffusion-based super-resolution, reducing computational costs while maintaining state-of-the-art quality by adaptively focusing on image textures and regions.

Contribution

The paper proposes the Time-Spatial-aware Sampling strategy (TSS) that leverages frequency and spatial domain insights to accelerate diffusion SR without additional training.

Findings

TSS achieves state-of-the-art performance with fewer iterations.

MUSIQ scores improve by 0.2 to 3.0 points.

Outperforms existing acceleration methods with half the steps.

Abstract

Diffusion models have gained attention for their success in modeling complex distributions, achieving impressive perceptual quality in SR tasks. However, existing diffusion-based SR methods often suffer from high computational costs, requiring numerous iterative steps for training and inference. Existing acceleration techniques, such as distillation and solver optimization, are generally task-agnostic and do not fully leverage the specific characteristics of low-level tasks like super-resolution (SR). In this study, we analyze the frequency- and spatial-domain properties of diffusion-based SR methods, revealing key insights into the temporal and spatial dependencies of high-frequency signal recovery. Specifically, high-frequency details benefit from concentrated optimization during early and late diffusion iterations, while spatially textured regions demand adaptive denoising strategies. Building on these observations, we propose the Time-Spatial-aware Sampling strategy (TSS) for the acceleration of Diffusion SR without any extra training cost. TSS combines Time Dynamic Sampling (TDS), which allocates more iterations to refining textures, and Spatial Dynamic Sampling (SDS), which dynamically adjusts strategies based on image content. Extensive evaluations across multiple benchmarks demonstrate that TSS achieves state-of-the-art (SOTA) performance with significantly fewer iterations, improving MUSIQ scores by 0.2 - 3.0 and outperforming the current acceleration methods with only half the number of steps.