Score Replacement with Bounded Deviation for Rare Prompt Generation

TL;DR

This paper introduces a score replacement method with bounded deviation for improved rare prompt generation in diffusion models, enabling adaptive switching between proxy and true scores to better synthesize rare concepts.

Contribution

It proposes a novel score replacement framework with a bounded deviation criterion, allowing adaptive control over prompt switching in diffusion models for rare concept generation.

Findings

Consistently improves rare concept synthesis across multiple models.

Outperforms strong baselines in automated metrics.

Enhances human evaluation scores.

Abstract

Diffusion models achieve impressive performance in high-fidelity image generation but often struggle with rare concepts that appear infrequently in the training distribution. Prior work attempts to address this issue by prompt switching, where generation begins with a frequent proxy prompt and later transitions to the original rare prompt. However, such designs typically rely on fixed schedules that disregard the model's internal dynamics, making them brittle across prompts and backbones. In this paper, we re-frame rare prompt generation through the lens of score replacement: the denoising trajectory of a rare prompt can be initially guided by the score of a semantically related frequent prompt, which acts as a proxy. However, as the process unfolds, the proxy score gradually diverges from the true rare prompt score. To control this drift, we introduce a bounded deviation criterion that triggers the switch once the deviation exceeds a threshold. This formulation offers both a principled justification and a practical mechanism for rare prompt generation, enabling adaptive switching that can be widely adopted by different models. Extensive experiments across SDXL, SD3, Flux, and Sana confirm that our method consistently improves rare concept synthesis, outperforming strong baselines in both automated metrics and human evaluations.