Token Perturbation Guidance for Diffusion Models

TL;DR

Token Perturbation Guidance (TPG) introduces a training-free, condition-agnostic method for diffusion models that enhances generation quality and alignment, matching the performance of classifier-free guidance without requiring architectural changes.

Contribution

The paper proposes TPG, a novel perturbation-based guidance technique that is training-free and applicable to both conditional and unconditional diffusion model generation.

Findings

Nearly 2× improvement in FID for unconditional generation.

Matches CFG in prompt alignment.

Applicable to various diffusion models without architectural changes.

Abstract

Classifier-free guidance (CFG) has become an essential component of modern diffusion models to enhance both generation quality and alignment with input conditions. However, CFG requires specific training procedures and is limited to conditional generation. To address these limitations, we propose Token Perturbation Guidance (TPG), a novel method that applies perturbation matrices directly to intermediate token representations within the diffusion network. TPG employs a norm-preserving shuffling operation to provide effective and stable guidance signals that improve generation quality without architectural changes. As a result, TPG is training-free and agnostic to input conditions, making it readily applicable to both conditional and unconditional generation. We further analyze the guidance term provided by TPG and show that its effect on sampling more closely resembles CFG compared to existing training-free guidance techniques. Extensive experiments on SDXL and Stable Diffusion 2.1 show that TPG achieves nearly a 2$\times$ improvement in FID for unconditional generation over the SDXL baseline, while closely matching CFG in prompt alignment. These results establish TPG as a general, condition-agnostic guidance method that brings CFG-like benefits to a broader class of diffusion models.