Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis

TL;DR

This paper introduces Noise Diffusion, a method leveraging large vision-language models to optimize initial noisy latents in diffusion models, significantly improving semantic faithfulness in generated images.

Contribution

It proposes a novel Noise Diffusion process guided by LVLMs, enhancing semantic alignment without altering model architectures or prompts.

Findings

Consistently improves semantic alignment across diffusion models

Theoretically analyzes conditions for effective semantic enhancement

Demonstrates adaptability and effectiveness in experiments

Abstract

Diffusion models have achieved impressive success in generating photorealistic images, but challenges remain in ensuring precise semantic alignment with input prompts. Optimizing the initial noisy latent offers a more efficient alternative to modifying model architectures or prompt engineering for improving semantic alignment. A latest approach, InitNo, refines the initial noisy latent by leveraging attention maps; however, these maps capture only limited information, and the effectiveness of InitNo is highly dependent on the initial starting point, as it tends to converge on a local optimum near this point. To this end, this paper proposes leveraging the language comprehension capabilities of large vision-language models (LVLMs) to guide the optimization of the initial noisy latent, and introduces the Noise Diffusion process, which updates the noisy latent to generate semantically faithful images while preserving distribution consistency. Furthermore, we provide a theoretical analysis of the condition under which the update improves semantic faithfulness. Experimental results demonstrate the effectiveness and adaptability of our framework, consistently enhancing semantic alignment across various diffusion models. The code is available at https://github.com/Bomingmiao/NoiseDiffusion.