A Noise is Worth Diffusion Guidance
Donghoon Ahn, Jiwon Kang, Sanghyun Lee, Jaewon Min, Minjae Kim,, Wooseok Jang, Hyoungwon Cho, Sayak Paul, SeonHwa Kim, Eunju Cha, Kyong Hwan, Jin, Seungryong Kim
TL;DR
This paper introduces a novel approach to diffusion models that eliminates the need for guidance by refining the initial noise, leading to high-quality image generation with improved efficiency and without guidance methods.
Contribution
Proposes \\ours, a noise refinement technique that replaces guidance in diffusion models, enabling guidance-free high-quality image synthesis with efficient noise-space learning.
Findings
Refined noise can produce high-quality images without guidance.
The method achieves rapid convergence with only 50K training pairs.
Eliminates the need for guidance, improving inference throughput and memory.
Abstract
Diffusion models excel in generating high-quality images. However, current diffusion models struggle to produce reliable images without guidance methods, such as classifier-free guidance (CFG). Are guidance methods truly necessary? Observing that noise obtained via diffusion inversion can reconstruct high-quality images without guidance, we focus on the initial noise of the denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we uncover that small low-magnitude low-frequency components significantly enhance the denoising process, removing the need for guidance and thus improving both inference throughput and memory. Expanding on this, we propose \ours, a novel method that replaces guidance methods with a single refinement of the initial noise. This refined noise enables high-quality image generation without guidance, within the same diffusion pipeline. Our…
Peer Reviews
Decision·ICLR 2026 Poster
1. Totally, it is well-written and easy to follow. 2. The overall motivation is clear. Optimizing the guidance to accelerate the inference of Diffusion models is useful. 3. The perspective from the initial noise is quite interesting. The idea of distilling guidance information into the initial noise, rather than into the denoising network itself, is also insightful and provides a fresh perspective on tackling the guidance overhead problem (with potential and compatibility for wider application i
1. A Special Form of Distillation: While the paper claims their method is "orthogonal" to guidance distillation, it is more accurately a clever variant rather than a new paradigm. The framework fits the knowledge distillation paradigm, the difference from guidance distillation is the locus of learning (the noise refiner $g_\phi$ v.s. the denoising network). This positioning overstates the fundamental novelty, more discussion/comparison/clarification should be made. 2. Loss of Critical Controllab
1. This idea is novel and plug-and-play. 2. It's orthogonal to other acceleration methods.
1. I'd like to know if optimizing Eq4 is difficult and costly, since it requires go through the network many times. 2. The result after refinement was still not good enough, significantly worse than the result generated by CFG. 3. For me, it's difficult to understand why noise-to-noise mapping has generalization properties. The paper lacks discussion and analysis on this aspect. For example, are the generated results of training samples has better quality than validation samples? For refined noi
The paper tackles an important topic in the diffusion community, which is related to guided generation. The main strength of this paper is in the idea it proposes. To the best of my knowledge, this is the first work to look into guidance distillation into the noise input, which seems like a simple yet brilliant idea. The execution of the method is also in itself very well done: the authors validate each contribution that make the method work using small experiments. The manuscript is easy to r
No major weaknesses, the appendix addresses many of my concerns already.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusic Technology and Sound Studies
MethodsDiffusion · Focus