EMAG: Self-Rectifying Diffusion Sampling with Exponential Moving Average Guidance

TL;DR

EMAG is a novel inference-time guidance method for diffusion models that adaptively selects attention layers to generate more challenging negatives, improving sample quality and human preference scores without additional training.

Contribution

EMAG introduces a training-free, adaptive layer-selection mechanism for diffusion transformers that enhances negative sample difficulty and complements existing guidance techniques.

Findings

EMAG improves human preference scores by +0.46 over CFG.

EMAG produces more semantically faithful, fine-grained negatives.

EMAG can be combined with other guidance methods for further improvements.

Abstract

In diffusion and flow-matching generative models, guidance techniques are widely used to improve sample quality and consistency. Classifier-free guidance (CFG) is the de facto choice in modern systems and achieves this by contrasting conditional and unconditional samples. Recent work explores contrasting negative samples at inference using a weaker model, via strong/weak model pairs, attention-based masking, stochastic block dropping, or perturbations to the self-attention energy landscape. While these strategies refine the generation quality, they still lack reliable control over the granularity or difficulty of the negative samples, and target-layer selection is often fixed. We propose Exponential Moving Average Guidance (EMAG), a training-free mechanism that modifies attention at inference time in diffusion transformers, with a statistics-based, adaptive layer-selection rule. Unlike prior methods, EMAG produces harder, semantically faithful negatives (fine-grained degradations), surfacing difficult failure modes, enabling the denoiser to refine subtle artifacts, boosting the quality and human preference score (HPS) by +0.46 over CFG. We further demonstrate that EMAG naturally composes with advanced guidance techniques, such as APG and CADS, further improving HPS.