Colorful-Noise: Training-Free Low-Frequency Noise Manipulation for Color-Based Conditional Image Generation

TL;DR

This paper introduces a training-free method to manipulate low-frequency noise in diffusion models, enabling control over global image structure and color in text-to-image generation.

Contribution

It reveals the role of low-frequency noise in diffusion models and proposes a simple, training-free technique to steer image structure and color without retraining.

Findings

Low-frequency noise primarily influences global structure and color.

Manipulating low-frequency noise effectively controls overall image appearance.

The method allows variability in fine details while maintaining desired global attributes.

Abstract

Text-to-image diffusion models generate images by gradually converting white Gaussian noise into a natural image. White Gaussian noise is well suited for producing diverse outputs from a single text prompt due to its absence of structure. However, this very property limits control over, and predictability of, specific visual attributes, as the noise is not human-interpretable. In this work, we investigate the characteristics of the input noise in diffusion models. We show that, although all frequencies in white Gaussian noise have comparable statistical energy, low-frequency components primarily determine the images global structure and color composition, while high-frequency components control finer details. Building on this observation, we demonstrate that simple manipulations of the low-frequency noise using low-frequency image priors can effectively condition the generation process to reconstruct these low-frequency visual cues. This allows us to define a simple, training-free method with minimal overhead that steers overall image structure and color, while letting high-frequency components freely emerge as fine details, enabling variability across generated outputs.