Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation

TL;DR

Diff-Aid introduces an inference-time adaptive method for text-to-image diffusion models that enhances semantic alignment and visual quality by dynamically adjusting interactions between textual and visual features across model stages.

Contribution

It proposes a flexible, plug-and-play inference-time approach that adaptively modulates text-image interactions, improving generation quality and interpretability in diffusion models.

Findings

Consistent improvements in prompt adherence and visual quality.

Enhanced human preference scores across experiments.

Effective integration with downstream applications like style transfer and zero-shot editing.

Abstract

Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions between textual and visual features. Prior approaches enhance such interactions via architectural design or handcrafted textual condition weighting, but lack flexibility and overlook the dynamic interactions across different blocks and denoising stages. To provide a more flexible and efficient solution to this problem, we propose Diff-Aid, a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps. Beyond improving generation quality, Diff-Aid yields interpretable modulation patterns that reveal how different blocks, timesteps, and textual tokens contribute to semantic alignment during denoising. As a plug-and-play module, Diff-Aid can be seamlessly integrated into downstream applications for further improvement, including style LoRAs, controllable generation, and zero-shot editing. Experiments on strong baselines (SD 3.5 and FLUX) demonstrate consistent improvements in prompt adherence, visual quality, and human preference across various metrics. Our code and models will be released.