Spectral Collapse in Diffusion Inversion

TL;DR

This paper identifies a spectral collapse problem in diffusion inversion for unpaired image translation, where spectral sparsity causes loss of detail, and proposes Orthogonal Variance Guidance to improve texture and structure preservation.

Contribution

The paper introduces Orthogonal Variance Guidance, a novel inference-time correction method that addresses spectral collapse in diffusion models for better image reconstruction.

Findings

OVG restores photorealistic textures effectively.

Spectral collapse causes oversmoothing in diffusion inversion.

OVG maintains structural fidelity while enhancing textures.

Abstract

Conditional diffusion inversion provides a powerful framework for unpaired image-to-image translation. However, we demonstrate through an extensive analysis that standard deterministic inversion (e.g. DDIM) fails when the source domain is spectrally sparse compared to the target domain (e.g., super-resolution, sketch-to-image). In these contexts, the recovered latent from the input does not follow the expected isotropic Gaussian distribution. Instead it exhibits a signal with lower frequencies, locking target sampling to oversmoothed and texture-poor generations. We term this phenomenon spectral collapse. We observe that stochastic alternatives attempting to restore the noise variance tend to break the semantic link to the input, leading to structural drift. To resolve this structure-texture trade-off, we propose Orthogonal Variance Guidance (OVG), an inference-time method that corrects the ODE dynamics to enforce the theoretical Gaussian noise magnitude within the null-space of the structural gradient. Extensive experiments on microscopy super-resolution (BBBC021) and sketch-to-image (Edges2Shoes) demonstrate that OVG effectively restores photorealistic textures while preserving structural fidelity.