A Non-negative VAE:the Generalized Gamma Belief Network

TL;DR

This paper introduces the Generalized Gamma Belief Network, a non-linear extension of the gamma belief network that enhances expressiveness and interpretability in deep latent variable models for text data.

Contribution

The paper proposes the Generalized GBN with a non-linear generative model and an upward-downward Weibull inference network, advancing beyond linear models for better expressiveness.

Findings

Outperforms state-of-the-art Gaussian VAEs in expressivity.

Achieves more interpretable and disentangled latent representations.

Demonstrates effectiveness on text data tasks.

Abstract

The gamma belief network (GBN), often regarded as a deep topic model, has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. Its notable capability to acquire interpretable latent factors is partially attributed to sparse and non-negative gamma-distributed latent variables. However, the existing GBN and its variations are constrained by the linear generative model, thereby limiting their expressiveness and applicability. To address this limitation, we introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. Since the parameters of the Generalized GBN no longer possess an analytic conditional posterior, we further propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables. The parameters of both the generative model and the inference network are jointly trained within the variational inference framework. Finally, we conduct comprehensive experiments on both expressivity and disentangled representation learning tasks to evaluate the performance of the Generalized GBN against state-of-the-art Gaussian variational autoencoders serving as baselines.