Let Triggers Control: Frequency-Aware Dropout for Effective Token Control

TL;DR

The paper introduces Frequency-Aware Dropout (FAD), a novel regularization technique that enhances trigger token controllability in text-to-image models by reducing co-occurrence entanglement without extra parameters.

Contribution

FAD is a simple, parameter-free dropout method that improves prompt controllability and personalization in diffusion models through co-occurrence analysis and curriculum-inspired scheduling.

Findings

FAD improves prompt fidelity and stylistic precision.

FAD enhances user-perceived quality in generated images.

FAD achieves these gains without additional parameters or architectural changes.

Abstract

Text-to-image models such as Stable Diffusion have achieved unprecedented levels of high-fidelity visual synthesis. As these models advance, personalization of generative models -- commonly facilitated through Low-Rank Adaptation (LoRA) with a dedicated trigger token -- has become a significant area of research. Previous works have naively assumed that fine-tuning with a single trigger token to represent new concepts. However, this often results in poor controllability, where the trigger token alone fails to reliably evoke the intended concept. We attribute this issue to the frequent co-occurrence of the trigger token with the surrounding context during fine-tuning, which entangles their representations and compromises the token's semantic distinctiveness. To disentangle this, we propose Frequency-Aware Dropout (FAD) -- a novel regularization technique that improves prompt controllability without adding new parameters. FAD consists of two key components: co-occurrence analysis and curriculum-inspired scheduling. Qualitative and quantitative analyses across token-based diffusion models (SD~1.5 and SDXL) and natural language--driven backbones (FLUX and Qwen-Image) demonstrate consistent gains in prompt fidelity, stylistic precision, and user-perceived quality. Our method provides a simple yet effective dropout strategy that enhances controllability and personalization in text-to-image generation. Notably, it achieves these improvements without introducing additional parameters or architectural modifications, making it readily applicable to existing models with minimal computational overhead.