ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generation

TL;DR

ADAPT is a training-free framework that improves rare concept generation in text-to-image diffusion models by deterministically planning prompt schedules using attention and orthogonal components, leading to more accurate and consistent results.

Contribution

It introduces a novel, training-free method for prompt scheduling that enhances rare concept synthesis by semantically aligning prompts with attention and orthogonal components.

Findings

Outperforms existing methods on the RareBench benchmark

Provides deterministic and precise control over rare concept generation

Enhances semantic accuracy without additional training

Abstract

Generating rare compositional concepts in text-to-image synthesis remains a challenge for diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for prompt scheduling, they suffer from inherent variance due to the randomness of language models and suboptimal guidance from iterative text embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity.