ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion

TL;DR

ScaleDiff is a versatile, training-free framework that enhances the resolution of pretrained diffusion models efficiently, utilizing Neighborhood Patch Attention, Latent Frequency Mixing, and Structure Guidance to improve image quality and detail.

Contribution

It introduces ScaleDiff, a model-agnostic, training-free method that significantly improves high-resolution image synthesis in diffusion models using novel attention and guidance techniques.

Findings

Achieves state-of-the-art results among training-free methods.

Improves image quality and inference speed.

Effective across U-Net and Diffusion Transformer architectures.

Abstract

Text-to-image diffusion models often exhibit degraded performance when generating images beyond their training resolution. Recent training-free methods can mitigate this limitation, but they often require substantial computation or are incompatible with recent Diffusion Transformer models. In this paper, we propose ScaleDiff, a model-agnostic and highly efficient framework for extending the resolution of pretrained diffusion models without any additional training. A core component of our framework is Neighborhood Patch Attention (NPA), an efficient mechanism that reduces computational redundancy in the self-attention layer with non-overlapping patches. We integrate NPA into an SDEdit pipeline and introduce Latent Frequency Mixing (LFM) to better generate fine details. Furthermore, we apply Structure Guidance to enhance global structure during the denoising process. Experimental results demonstrate that ScaleDiff achieves state-of-the-art performance among training-free methods in terms of both image quality and inference speed on both U-Net and Diffusion Transformer architectures.