Neural Bayes: A Generic Parameterization Method for Unsupervised Representation Learning

TL;DR

Neural Bayes introduces a flexible parameterization technique enabling closed-form computation of complex statistical quantities, facilitating new objectives in unsupervised representation learning such as mutual information maximization and manifold clustering.

Contribution

It provides a novel neural network-based parameterization method that allows exact computation of distributions and mutual information, advancing unsupervised learning and clustering methods.

Findings

Closed-form mutual information computation improves representation learning.

Effective disjoint manifold clustering demonstrates the method's practical utility.

The approach enables new unsupervised learning objectives with theoretical guarantees.

Abstract

We introduce a parameterization method called Neural Bayes which allows computing statistical quantities that are in general difficult to compute and opens avenues for formulating new objectives for unsupervised representation learning. Specifically, given an observed random variable $\mathbf{x}$ and a latent discrete variable $z$, we can express $p(\mathbf{x}|z)$, $p(z|\mathbf{x})$ and $p(z)$ in closed form in terms of a sufficiently expressive function (Eg. neural network) using our parameterization without restricting the class of these distributions. To demonstrate its usefulness, we develop two independent use cases for this parameterization: 1. Mutual Information Maximization (MIM): MIM has become a popular means for self-supervised representation learning. Neural Bayes allows us to compute mutual information between observed random variables $\mathbf{x}$ and latent discrete random variables $z$ in closed form. We use this for learning image representations and show its usefulness on downstream classification tasks. 2. Disjoint Manifold Labeling: Neural Bayes allows us to formulate an objective which can optimally label samples from disjoint manifolds present in the support of a continuous distribution. This can be seen as a specific form of clustering where each disjoint manifold in the support is a separate cluster. We design clustering tasks that obey this formulation and empirically show that the model optimally labels the disjoint manifolds. Our code is available at \url{https://github.com/salesforce/NeuralBayes}