Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation

TL;DR

This paper introduces Adaptive Auxiliary Prompt Blending (AAPB), a training-free method that improves the fidelity and semantic accuracy of diffusion-based text-to-image models, especially for rare concepts and editing tasks.

Contribution

AAPB provides a novel, closed-form, adaptive prompt blending framework grounded in Tweedie's identity, enhancing target-faithful diffusion generation without additional training.

Findings

Improves semantic accuracy on RareBench dataset

Enhances structural fidelity on FlowEdit dataset

Outperforms prior training-free baselines

Abstract

Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) - a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.