Latent Swap Joint Diffusion for 2D Long-Form Latent Generation

TL;DR

This paper presents Swap Forward (SaFa), a novel latent swap joint diffusion method that enhances long-form 2D and audio generation by improving spectrum coherence and cross-view consistency, outperforming existing methods in quality and speed.

Contribution

The paper introduces Self-Loop Latent Swap and Reference-Guided Latent Swap techniques to address spectrum aliasing and global consistency in joint diffusion, advancing long-form multi-view generation.

Findings

SaFa outperforms existing joint diffusion methods in audio quality.

SaFa achieves comparable panorama generation with 2-20x faster speed.

SaFa demonstrates better high-frequency preservation and view consistency.

Abstract

This paper introduces Swap Forward (SaFa), a modality-agnostic and efficient method to generate seamless and coherence long spectrum and panorama through latent swap joint diffusion across multi-views. We first investigate the spectrum aliasing problem in spectrum-based audio generation caused by existing joint diffusion methods. Through a comparative analysis of the VAE latent representation of Mel-spectra and RGB images, we identify that the failure arises from excessive suppression of high-frequency components during the spectrum denoising process due to the averaging operator. To address this issue, we propose Self-Loop Latent Swap, a frame-level bidirectional swap applied to the overlapping region of adjacent views. Leveraging stepwise differentiated trajectories of adjacent subviews, this swap operator adaptively enhances high-frequency components and avoid spectrum distortion. Furthermore, to improve global cross-view consistency in non-overlapping regions, we introduce Reference-Guided Latent Swap, a unidirectional latent swap operator that provides a centralized reference trajectory to synchronize subview diffusions. By refining swap timing and intervals, we can achieve a cross-view similarity-diversity balance in a forward-only manner. Quantitative and qualitative experiments demonstrate that SaFa significantly outperforms existing joint diffusion methods and even training-based methods in audio generation using both U-Net and DiT models, along with effective longer length adaptation. It also adapts well to panorama generation, achieving comparable performance with 2 $\sim$ 20 $\times$ faster speed and greater model generalizability. More generation demos are available at https://swapforward.github.io/