Learning Sparse Latent Representations for Generator Model

TL;DR

This paper introduces an unsupervised method for learning sparse latent representations in generator models using a spike and slab prior, improving interpretability, robustness, and efficiency in image tasks.

Contribution

The paper proposes a novel unsupervised approach with a spike and slab prior to enforce sparsity in generator latent spaces, enhancing interpretability and robustness.

Findings

Preserves most information with sparse latent codes

Improves classification and denoising robustness

Learns disentangled, explainable semantics

Abstract

Sparsity is a desirable attribute. It can lead to more efficient and more effective representations compared to the dense model. Meanwhile, learning sparse latent representations has been a challenging problem in the field of computer vision and machine learning due to its complexity. In this paper, we present a new unsupervised learning method to enforce sparsity on the latent space for the generator model with a gradually sparsified spike and slab distribution as our prior. Our model consists of only one top-down generator network that maps the latent variable to the observed data. Latent variables can be inferred following generator posterior direction using non-persistent gradient based method. Spike and Slab regularization in the inference step can push non-informative latent dimensions towards zero to induce sparsity. Extensive experiments show the model can preserve majority of the information from original images with sparse representations while demonstrating improved results compared to other existing methods. We observe that our model can learn disentangled semantics and increase explainability of the latent codes while boosting the robustness in the task of classification and denoising.