Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization

TL;DR

This paper investigates the impact of noise selection and optimization on diffusion models, revealing that choosing and enhancing noise based on inversion stability significantly improves data generation quality without model fine-tuning.

Contribution

It introduces the first noise selection and optimization methods based on inversion stability, improving diffusion model outputs without requiring fine-tuning.

Findings

Noise selection based on inversion stability improves quality.

Noise optimization enhances generated results without fine-tuning.

Significant improvements in human preference and evaluation metrics.

Abstract

Diffusion models that can generate high-quality data from randomly sampled Gaussian noises have become the mainstream generative method in both academia and industry. Are randomly sampled Gaussian noises equally good for diffusion models? While a large body of works tried to understand and improve diffusion models, previous works overlooked the possibility to select or optimize the sampled noise the possibility of selecting or optimizing sampled noises for improving diffusion models. In this paper, we mainly made three contributions. First, we report that not all noises are created equally for diffusion models. We are the first to hypothesize and empirically observe that the generation quality of diffusion models significantly depend on the noise inversion stability. This naturally provides us a noise selection method according to the inversion stability. Second, we further propose a novel noise optimization method that actively enhances the inversion stability of arbitrary given noises. Our method is the first one that works on noise space to generally improve generated results without fine-tuning diffusion models. Third, our extensive experiments demonstrate that the proposed noise selection and noise optimization methods both significantly improve representative diffusion models, such as SDXL and SDXL-turbo, in terms of human preference and other objective evaluation metrics. For example, the human preference winning rates of noise selection and noise optimization over the baselines can be up to 57% and 72.5%, respectively, on DrawBench.