Geometry-Aware Attention Guidance for Diffusion Models via Modern Hopfield Dynamics

TL;DR

This paper introduces Geometry-Aware Attention Guidance (GAG), a novel, training-free method that enhances diffusion model sampling efficiency and quality by leveraging Modern Hopfield dynamics and attention extrapolation.

Contribution

It provides a theoretical analysis of attention extrapolation, proposes a universal, plug-and-play guidance method, and demonstrates consistent improvements across multiple architectures and sampling regimes.

Findings

GAG improves generation quality across diverse backbones.

The method generalizes to multiple architectures and regimes.

It offers a unified perspective via Anderson Acceleration interpretation.

Abstract

Classifier-Free Guidance (CFG) improves sample quality in diffusion models, but its dual-pass inference and reliance on null-condition training limit its use in few-step regimes. Attention-space guidance has emerged as a complementary paradigm that addresses this gap, yet why prior sparse-vs-dense attention guidance works remains elusive. We address this by analyzing attention extrapolation through Modern Hopfield dynamics, proving two directional properties of the sparse-dense discrepancy under shared conditioning that together certify it as a directionally consistent acceleration signal. Building on this, we propose Geometry-Aware Attention Guidance (GAG), a training-free, plug-and-play extrapolation rule that decomposes the discrepancy into parallel and orthogonal components relative to the retrieval direction, amplifying the convergence-aligned component while suppressing off-manifold noise; stability follows from a weak contraction property. We further provide an interpretation of this extrapolation as first-order Anderson Acceleration in attention space, offering a unified perspective on attention extrapolation methods. GAG is a universal method that generalizes across architectures (UNet, MMDiT) and sampling regimes (multi-step, few-step), consistently improving generation quality on diverse backbones, including FLUX.1, the recent FLUX.2, and Qwen-Image, with minimal computational overhead.