Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space

TL;DR

This paper identifies low-frequency drift in VAE latent space as a key cause of semantic drift in multi-turn diffusion transformer image editing and proposes a plug-and-play alignment method to mitigate it.

Contribution

The authors introduce VAE-LFA, a training-free, plug-and-play low-frequency alignment technique that reduces semantic drift in multi-turn diffusion transformer editing.

Findings

VAE-LFA significantly improves semantic consistency in multi-turn editing.

The method is effective for both white-box and black-box diffusion models.

VAE-LFA preserves high-frequency details while suppressing low-frequency drift.

Abstract

Recent advances in diffusion transformers (DiTs) have enabled promising single-turn image editing capabilities. However, multi-turn editing often leads to progressive semantic drift and quality degradation.In this work, we study this problem from a latent-space frequency perspective by decomposing the editing process into two functional components: VAE and DiT. Through systematic analysis in the VAE latent space, we uncover that the DiT introduces dominant low-frequency drift that accumulates as semantic misalignment across editing rounds, while the VAE contributes comparatively stable reconstruction bias.Based on this insight, we propose VAE-LFA (Low Frequency Alignment), a training-free, plug-and-play method that performs alignment in VAE latent space. VAE-LFA decomposes latent discrepancies across editing rounds via low-pass filtering, and aligns low-frequency statistics to an exponential moving average of previous rounds, effectively suppressing accumulated semantic drift while preserving high-frequency details.Our method requires no retraining, ground-truth priors, or access to diffusion parameters, making it applicable to both white-box and black-box DiT editors. For white-box models, VAE-LFA is seamlessly integrated into the editing pipeline by eliminating redundant VAE round trips; for black-box models, it operates via an off-the-shelf VAE to perform inter-round latent alignment.Extensive experiments demonstrate that VAE-LFA improves semantic consistency and visual fidelity across diverse multi-turn editing scenarios, including both controlled and in-the-wild images.