Latent-Compressed Variational Autoencoder for Video Diffusion Models

TL;DR

This paper introduces a latent compression technique for video VAEs that removes high-frequency components to improve reconstruction quality and model performance without reducing latent channels.

Contribution

A novel latent compression method that preserves reconstruction fidelity by removing high-frequency components, enhancing video VAE performance.

Findings

Achieves better video reconstruction quality than strong baselines.

Maintains the same compression ratio while improving performance.

Addresses issues caused by excessive latent channels in diffusion models.

Abstract

Video variational autoencoders (VAEs) used in latent diffusion models typically require a sufficiently large number of latent channels to ensure high-quality video reconstruction. However, recent studies have revealed that an excessive number of latent channels can impede the convergence of latent diffusion models and deteriorate their generative performance, even when reconstruction quality remains high. We propose a latent compression method that removes high-frequency components in video latent representations rather than directly reducing the number of channels, which often compromises reconstruction fidelity. Experimental results demonstrate that the proposed method achieves superior video reconstruction quality compared to strong baselines while maintaining the same overall compression ratio.