Boosting Latent Diffusion Models via Disentangled Representation Alignment

TL;DR

This paper introduces Send-VAE, a novel semantic-disentangled VAE that improves latent space structure for high-quality image generation by aligning local features with dense semantics, outperforming previous methods.

Contribution

Send-VAE employs a non-linear mapping to enhance semantic disentanglement in VAEs, addressing limitations of shallow alignment strategies for latent diffusion models.

Findings

Achieves state-of-the-art FID of 1.21 on ImageNet 256x256.

Demonstrates improved attribute separability in VAE latent spaces.

Establishes a new evaluation paradigm for VAE latent representations.

Abstract

Latent Diffusion Models (LDMs) rely heavily on the compressed latent space provided by Variational Autoencoders (VAEs) for high-quality image generation. Recent studies have attempted to obtain generation-friendly VAEs by directly adopting alignment strategies from LDM training, leveraging Vision Foundation Models (VFMs) as representation alignment targets. However, such alignment paradigms overlook the fundamental differences in representational requirements between LDMs and VAEs. Simple feature mapping from local patches to high-dimensional semantics can induce semantic collapse, leading to the loss of fine-grained attributes. In this paper, we reveal a key insight: unlike LDMs that benefit from high-level global semantics, a generation-friendly VAE must possess strong semantic disentanglement capabilities to preserve fine-grained, attribute-level information in a structured manner. To address this discrepancy, we propose the Semantic-Disentangled VAE (Send-VAE). Deviating from previous shallow alignment approaches, Send-VAE introduces a non-linear mapping architecture to effectively bridge the local structures of VAEs and the dense semantics of VFMs, thereby encouraging emergent disentangled properties in the latent space without explicit regularization. Extensive experiments establish a new paradigm for evaluating VAE latent spaces via low-level attribute separability and demonstrate that Send-VAE achieves state-of-the-art generation quality (FID of 1.21) on ImageNet 256x256.