Reliable Categorical Variational Inference with Mixture of Discrete Normalizing Flows

TL;DR

This paper introduces a new differentiable reparameterization for categorical variables using a mixture of discrete normalizing flows, enabling more accurate and stable variational inference with discrete latent variables.

Contribution

It proposes a novel mixture of discrete normalizing flows for categorical distributions, providing a valid probability mass and improved optimization stability over relaxations.

Findings

Defines a proper discrete distribution for categorical variables.

Enables direct optimization of the evidence lower bound.

Less sensitive to relaxation hyperparameters.

Abstract

Variational approximations are increasingly based on gradient-based optimization of expectations estimated by sampling. Handling discrete latent variables is then challenging because the sampling process is not differentiable. Continuous relaxations, such as the Gumbel-Softmax for categorical distribution, enable gradient-based optimization, but do not define a valid probability mass for discrete observations. In practice, selecting the amount of relaxation is difficult and one needs to optimize an objective that does not align with the desired one, causing problems especially with models having strong meaningful priors. We provide an alternative differentiable reparameterization for categorical distribution by composing it as a mixture of discrete normalizing flows. It defines a proper discrete distribution, allows directly optimizing the evidence lower bound, and is less sensitive to the hyperparameter controlling relaxation.