Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector

TL;DR

This paper introduces a guidance method for diffusion models that uses a representation alignment projector to improve semantic consistency during image generation, without altering the model architecture.

Contribution

It proposes a novel guidance scheme with a representation alignment projector that enhances semantic alignment in diffusion models during inference.

Findings

Significant reduction in FID scores on ImageNet synthesis tasks.

Outperforms existing guidance methods in semantic coherence and visual fidelity.

Provides complementary improvements when combined with classifier-free guidance.

Abstract

Recent progress in generative modeling has enabled high-quality visual synthesis with diffusion-based frameworks, supporting controllable sampling and large-scale training. Inference-time guidance methods such as classifier-free and representative guidance enhance semantic alignment by modifying sampling dynamics; however, they do not fully exploit unsupervised feature representations. Although such visual representations contain rich semantic structure, their integration during generation is constrained by the absence of ground-truth reference images at inference. This work reveals semantic drift in the early denoising stages of diffusion transformers, where stochasticity results in inconsistent alignment even under identical conditioning. To mitigate this issue, we introduce a guidance scheme using a representation alignment projector that injects representations predicted by a projector into intermediate sampling steps, providing an effective semantic anchor without modifying the model architecture. Experiments on SiTs and REPAs show notable improvements in class-conditional ImageNet synthesis, achieving substantially lower FID scores; for example, REPA-XL/2 improves from 5.9 to 3.3, and the proposed method outperforms representative guidance when applied to SiT models. The approach further yields complementary gains when combined with classifier-free guidance, demonstrating enhanced semantic coherence and visual fidelity. These results establish representation-informed diffusion sampling as a practical strategy for reinforcing semantic preservation and image consistency.