The Universal Normal Embedding

TL;DR

This paper proposes the Universal Normal Embedding hypothesis, suggesting a shared Gaussian latent space underlying generative models and vision encoders, supported by empirical evidence from a new dataset and enabling controllable image edits.

Contribution

It introduces NoiseZoo, a dataset linking diffusion noise and encoder embeddings, and demonstrates that both can be modeled as noisy linear projections of a shared Gaussian latent space.

Findings

Strong attribute prediction from linear probes in both spaces.

Controllable image edits via linear directions without architectural changes.

Empirical support for the shared Gaussian latent space hypothesis.

Abstract

Generative models and vision encoders have largely advanced on separate tracks, optimized for different goals and grounded in different mathematical principles. Yet, they share a fundamental property: latent space Gaussianity. Generative models map Gaussian noise to images, while encoders map images to semantic embeddings whose coordinates empirically behave as Gaussian. We hypothesize that both are views of a shared latent source, the Universal Normal Embedding (UNE): an approximately Gaussian latent space from which encoder embeddings and DDIM-inverted noise arise as noisy linear projections. To test our hypothesis, we introduce NoiseZoo, a dataset of per-image latents comprising DDIM-inverted diffusion noise and matching encoder representations (CLIP, DINO). On CelebA, linear probes in both spaces yield strong, aligned attribute predictions, indicating that generative noise encodes meaningful semantics along linear directions. These directions further enable faithful, controllable edits (e.g., smile, gender, age) without architectural changes, where simple orthogonalization mitigates spurious entanglements. Taken together, our results provide empirical support for the UNE hypothesis and reveal a shared Gaussian-like latent geometry that concretely links encoding and generation. Code and data are available https://rbetser.github.io/UNE/