LENS: Low-Frequency Eigen Noise Shaping for Efficient Diffusion Sampling

TL;DR

LENS is an efficient noise modulation framework for diffusion models that operates in a low-dimensional subspace, significantly reducing computational costs while maintaining high image quality.

Contribution

LENS introduces a low-frequency eigen noise shaping method that modulates noise in a low-dimensional space, improving efficiency in diffusion sampling.

Findings

Reduces FLOPs by 400-700 times compared to prior methods.

Cuts model parameters by 25-75 times.

Decreases inference overhead by 10-20 times.

Abstract

Distilled diffusion models accelerate image generation by reducing the number of denoising steps, but often suffer from degraded image quality. To mitigate this trade-off, test-time optimization methods improve quality, yet their iterative nature incurs substantial computational overhead and leads to slow inference, limiting practical usability. Recent hypernetwork-based approaches amortize this process during training, but still require costly noise modulation in high-dimensional latent spaces. In this work, we propose LENS (Low-frequency Eigen Noise Shaping), an efficient noise modulation framework that operates in a low-dimensional subspace. Our approach is motivated by the observation that low-frequency components of the noise largely determine the global structure and visual fidelity of generated images. Based on this observation, we provide a theoretical justification for restricting modulation to the low-frequency subspace and derive a principled training objective. Building on this, LENS employs a lightweight, standalone network to selectively modulate these components, enabling efficient and targeted noise modulation. Extensive experiments demonstrate that LENS achieves competitive image quality while reducing FLOPs by 400-700$\times$, model parameters by 25-75$\times$, and inference-time overhead by 10-20$\times$ compared to prior methods.