It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Models

TL;DR

This paper proposes a noise optimization method to enhance diversity and reduce mode collapse in trained diffusion models, improving generation quality without sacrificing fidelity.

Contribution

It introduces a simple noise optimization approach and analyzes noise frequency profiles to better mitigate mode collapse in diffusion models.

Findings

Noise optimization improves diversity in generated images.

Alternative noise initializations with different frequency profiles enhance results.

The method outperforms existing guidance techniques in quality and diversity.

Abstract

Contemporary text-to-image models exhibit a surprising degree of mode collapse, as can be seen when sampling several images given the same text prompt. Previous work has attempted to address this issue by steering the model using guidance mechanisms, or by generating a large pool of candidates and refining them. In this work, we take a different direction and aim for diversity in generations via noise optimization. Specifically, we show that a simple noise optimization objective can mitigate mode collapse while preserving the fidelity of the base model. We also analyze the frequency characteristics of the noise and show that alternative noise initializations with different frequency profiles can improve both optimization and search. Our experiments demonstrate that noise optimization yields superior results in terms of generation quality and diversity.