Beyond the Noise: Aligning Prompts with Latent Representations in Diffusion Models

TL;DR

This paper introduces NoisyCLIP, a method for early detection of text/image misalignment during diffusion model image generation, reducing computational costs and enabling real-time quality control.

Contribution

It is the first to benchmark prompt-to-latent misalignment detection during diffusion process using dual encoders, achieving high alignment accuracy with lower computational costs.

Findings

Reduces computational cost by 50%

Achieves 98% of CLIP alignment performance

Enables real-time alignment assessment during generation

Abstract

Conditional diffusion models rely on language-to-image alignment methods to steer the generation towards semantically accurate outputs. Despite the success of this architecture, misalignment and hallucinations remain common issues and require automatic misalignment detection tools to improve quality, for example by applying them in a Best-of-N (BoN) post-generation setting. Unfortunately, measuring the alignment after the generation is an expensive step since we need to wait for the overall generation to finish to determine prompt adherence. In contrast, this work hypothesizes that text/image misalignments can be detected early in the denoising process, enabling real-time alignment assessment without waiting for the complete generation. In particular, we propose NoisyCLIP a method that measures semantic alignment in the noisy latent space. This work is the first to explore and benchmark prompt-to-latent misalignment detection during image generation using dual encoders in the reverse diffusion process. We evaluate NoisyCLIP qualitatively and quantitatively and find it reduces computational cost by 50% while achieving 98% of CLIP alignment performance in BoN settings. This approach enables real-time alignment assessment during generation, reducing costs without sacrificing semantic fidelity.