Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation

TL;DR

Latent Forcing introduces a method that combines the efficiency of latent diffusion with raw image processing by reordering the denoising trajectory, enabling high-quality pixel-level image generation.

Contribution

It proposes Latent Forcing, a novel approach that jointly processes latents and pixels with tuned noise schedules, improving diffusion models' performance on raw images.

Findings

Achieves state-of-the-art results on ImageNet for pixel diffusion.

Demonstrates the importance of conditioning order in diffusion models.

Analyzes the relationship between tokenizer quality and diffusability.

Abstract

Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, conditional versus unconditional generation, and how tokenizer reconstruction quality relates to diffusability. Applied to ImageNet, Latent Forcing achieves a new state-of-the-art for diffusion transformer-based pixel generation at our compute scale.