A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models

TL;DR

This paper introduces Noise PPO, a minimalist reinforcement learning method that fine-tunes text-to-image diffusion models by optimizing initial noise, leading to improved alignment and sample quality without complex procedures.

Contribution

The paper proposes Noise PPO, a simple RL approach that fine-tunes diffusion models by learning a prompt-conditioned initial noise generator, avoiding trajectory storage and complex guidance.

Findings

Optimizing initial noise improves image-text alignment.

Significant gains at low inference steps.

Benefits diminish with more inference steps.

Abstract

Recent work uses reinforcement learning (RL) to fine-tune text-to-image diffusion models, improving text-image alignment and sample quality. However, existing approaches introduce unnecessary complexity: they cache the full sampling trajectory, depend on differentiable reward models or large preference datasets, or require specialized guidance techniques. Motivated by the "golden noise" hypothesis -- that certain initial noise samples can consistently yield superior alignment -- we introduce Noise PPO, a minimalist RL algorithm that leaves the pre-trained diffusion model entirely frozen and learns a prompt-conditioned initial noise generator. Our approach requires no trajectory storage, reward backpropagation, or complex guidance tricks. Extensive experiments show that optimizing the initial noise distribution consistently improves alignment and sample quality over the original model, with the most significant gains at low inference steps. As the number of inference steps increases, the benefit of noise optimization diminishes but remains present. These findings clarify the scope and limitations of the golden noise hypothesis and reinforce the practical value of minimalist RL fine-tuning for diffusion models.