Exploring Representation-Aligned Latent Space for Better Generation

TL;DR

This paper introduces ReaLS, a semantic prior integration method for latent diffusion models, significantly improving generation quality and enabling better downstream task performance.

Contribution

ReaLS is a novel approach that aligns latent space with semantic priors, enhancing generation quality and downstream task capabilities in diffusion models.

Findings

15% improvement in FID metric with ReaLS

Enhanced performance in segmentation and depth estimation tasks

ReaLS improves the quality of latent representations in diffusion models

Abstract

Generative models serve as powerful tools for modeling the real world, with mainstream diffusion models, particularly those based on the latent diffusion model paradigm, achieving remarkable progress across various tasks, such as image and video synthesis. Latent diffusion models are typically trained using Variational Autoencoders (VAEs), interacting with VAE latents rather than the real samples. While this generative paradigm speeds up training and inference, the quality of the generated outputs is limited by the latents' quality. Traditional VAE latents are often seen as spatial compression in pixel space and lack explicit semantic representations, which are essential for modeling the real world. In this paper, we introduce ReaLS (Representation-Aligned Latent Space), which integrates semantic priors to improve generation performance. Extensive experiments show that fundamental DiT and SiT trained on ReaLS can achieve a 15% improvement in FID metric. Furthermore, the enhanced semantic latent space enables more perceptual downstream tasks, such as segmentation and depth estimation.