FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling

TL;DR

FreCaS introduces a frequency-aware cascaded sampling framework that significantly improves high-resolution image generation efficiency and quality by progressively expanding frequency bands and refining details in diffusion models.

Contribution

The paper proposes a novel frequency-aware cascaded sampling method with guidance strategy and cross-attention fusion, enabling faster and higher-quality high-resolution image synthesis.

Findings

FreCaS outperforms state-of-the-art methods in image quality and speed.

FreCaS is approximately 2.86x faster than ScaleCrafter and 6.07x faster than DemoFusion.

FreCaS achieves notable FID improvements on 2048x2048 images.

Abstract

While image generation with diffusion models has achieved a great success, generating images of higher resolution than the training size remains a challenging task due to the high computational cost. Current methods typically perform the entire sampling process at full resolution and process all frequency components simultaneously, contradicting with the inherent coarse-to-fine nature of latent diffusion models and wasting computations on processing premature high-frequency details at early diffusion stages. To address this issue, we introduce an efficient $\textbf{Fre}$quency-aware $\textbf{Ca}$scaded $\textbf{S}$ampling framework, $\textbf{FreCaS}$ in short, for higher-resolution image generation. FreCaS decomposes the sampling process into cascaded stages with gradually increased resolutions, progressively expanding frequency bands and refining the corresponding details. We propose an innovative frequency-aware classifier-free guidance (FA-CFG) strategy to assign different guidance strengths for different frequency components, directing the diffusion model to add new details in the expanded frequency domain of each stage. Additionally, we fuse the cross-attention maps of previous and current stages to avoid synthesizing unfaithful layouts. Experiments demonstrate that FreCaS significantly outperforms state-of-the-art methods in image quality and generation speed. In particular, FreCaS is about 2.86$\times$ and 6.07$\times$ faster than ScaleCrafter and DemoFusion in generating a 2048$\times$2048 image using a pre-trained SDXL model and achieves an FID$_b$ improvement of 11.6 and 3.7, respectively. FreCaS can be easily extended to more complex models such as SD3. The source code of FreCaS can be found at https://github.com/xtudbxk/FreCaS.