Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspective

TL;DR

This paper identifies the frequency-related challenges in high-dimensional latent spaces for diffusion models and introduces FreqWarm, a curriculum that improves generation quality without retraining autoencoders.

Contribution

We propose FreqWarm, a frequency warm-up curriculum that enhances high-frequency exposure during training, improving diffusion-based generation quality across various autoencoders.

Findings

FreqWarm reduces gFID scores significantly across multiple autoencoders.

High-frequency latent components are crucial for detailed generation.

Managing frequency exposure improves diffusibility of high-dimensional latent spaces.

Abstract

Latent diffusion has become the default paradigm for visual generation, yet we observe a persistent reconstruction-generation trade-off as latent dimensionality increases: higher-capacity autoencoders improve reconstruction fidelity but generation quality eventually declines. We trace this gap to the different behaviors in high-frequency encoding and decoding. Through controlled perturbations in both RGB and latent domains, we analyze encoder/decoder behaviors and find that decoders depend strongly on high-frequency latent components to recover details, whereas encoders under-represent high-frequency contents, yielding insufficient exposure and underfitting in high-frequency bands for diffusion model training. To address this issue, we introduce FreqWarm, a plug-and-play frequency warm-up curriculum that increases early-stage exposure to high-frequency latent signals during diffusion or flow-matching training -- without modifying or retraining the autoencoder. Applied across several high-dimensional autoencoders, FreqWarm consistently improves generation quality: decreasing gFID by 14.11 on Wan2.2-VAE, 6.13 on LTX-VAE, and 4.42 on DC-AE-f32, while remaining architecture-agnostic and compatible with diverse backbones. Our study shows that explicitly managing frequency exposure can successfully turn high-dimensional latent spaces into more diffusible targets.