LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling

TL;DR

This paper introduces a novel sparse, dictionary-based latent space representation for generative models, which improves expressiveness and reconstruction quality over traditional vector quantization methods by relaxing the assumption of inherent data discreteness.

Contribution

The authors propose a union of subspaces model with learned dictionaries and sparsity constraints for latent space representation, enhancing generative modeling performance.

Findings

Sparse latent representations outperform VQ in reconstruction quality.

The approach addresses codebook collapse issues in VQ models.

Latent space sparsity offers benefits beyond discretization, such as better expressiveness.

Abstract

Learning compact and meaningful latent space representations has been shown to be very useful in generative modeling tasks for visual data. One particular example is applying Vector Quantization (VQ) in variational autoencoders (VQ-VAEs, VQ-GANs, etc.), which has demonstrated state-of-the-art performance in many modern generative modeling applications. Quantizing the latent space has been justified by the assumption that the data themselves are inherently discrete in the latent space (like pixel values). In this paper, we propose an alternative representation of the latent space by relaxing the structural assumption than the VQ formulation. Specifically, we assume that the latent space can be approximated by a union of subspaces model corresponding to a dictionary-based representation under a sparsity constraint. The dictionary is learned/updated during the training process. We apply this approach to look at two models: Dictionary Learning Variational Autoencoders (DL-VAEs) and DL-VAEs with Generative Adversarial Networks (DL-GANs). We show empirically that our more latent space is more expressive and has leads to better representations than the VQ approach in terms of reconstruction quality at the expense of a small computational overhead for the latent space computation. Our results thus suggest that the true benefit of the VQ approach might not be from discretization of the latent space, but rather the lossy compression of the latent space. We confirm this hypothesis by showing that our sparse representations also address the codebook collapse issue as found common in VQ-family models.