Determinism of Randomness: Prompt-Residual Seed Shaping for Diffusion Generation

TL;DR

This paper explores the seed sensitivity in diffusion models, providing a geometric explanation and introducing a training-free seed-shaping method that improves generation quality and alignment.

Contribution

It offers a geometric perspective on seed sensitivity and proposes a novel seed-shaping technique using a prompt residual as a proxy, enhancing diffusion model outputs.

Findings

The seed sensitivity is linked to a semantic map with a degenerate pullback semi-metric.

The proposed seed-shaping method improves alignment and quality metrics.

The method is effective across multiple generation benchmarks.

Abstract

Diffusion models start generation from an isotropic Gaussian latent, yet changing only the random seed can lead to large differences in prompt faithfulness, composition, and visual quality. We study this seed sensitivity through the semantic map from initial noise to generated meaning. Although the sampling flow is locally invertible, the subsequent semantic projection is many-to-one, inducing a degenerate pullback semi-metric on the latent space: most local directions are nearly semantic-invariant, while semantic-sensitive variation is concentrated in a much smaller horizontal subspace. This provides an explanatory geometric view of the seed lottery. Motivated by this view, we introduce a training-free prompt-residual seed-shaping procedure. Rather than claiming to recover the exact horizontal space, the method uses a single high-noise cold-start prompt residual as a model-coupled proxy, injects only its tangential component, and retracts the seed to the original Gaussian radius shell. This keeps the initialization prior-compatible while adding only one conditional/unconditional probe before standard sampling. Across multiple generation benchmarks, the method improves alignment and quality metrics over standard sampling, supporting both the practical value of the proxy and the explanatory relevance of semantic anisotropy.