Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation

TL;DR

This paper introduces Null-TTA, a novel test-time alignment method for text-to-image diffusion models that optimizes unconditional embeddings in semantic space, achieving state-of-the-art alignment without reward hacking.

Contribution

Null-TTA is the first approach to align diffusion models by optimizing in semantic space, ensuring coherent alignment and preventing reward hacking during test time.

Findings

Null-TTA achieves state-of-the-art test-time alignment performance.

Null-TTA maintains strong generalization across different rewards.

Null-TTA prevents reward hacking by optimizing in semantic space.

Abstract

Test-time alignment (TTA) aims to adapt models to specific rewards during inference. However, existing methods tend to either under-optimise or over-optimise (reward hack) the target reward function. We propose Null-Text Test-Time Alignment (Null-TTA), which aligns diffusion models by optimising the unconditional embedding in classifier-free guidance, rather than manipulating latent or noise variables. Due to the structured semantic nature of the text embedding space, this ensures alignment occurs on a semantically coherent manifold and prevents reward hacking (exploiting non-semantic noise patterns to improve the reward). Since the unconditional embedding in classifier-free guidance serves as the anchor for the model's generative distribution, Null-TTA directly steers model's generative distribution towards the target reward rather than just adjusting the samples, even without updating model parameters. Thanks to these desirable properties, we show that Null-TTA achieves state-of-the-art target test-time alignment while maintaining strong cross-reward generalisation. This establishes semantic-space optimisation as an effective and principled novel paradigm for TTA.