Bayesian neural networks and dimensionality reduction

TL;DR

This paper explores Bayesian neural networks with MCMC for better uncertainty quantification in non-linear dimensionality reduction, highlighting challenges and proposing solutions with practical demonstrations.

Contribution

It introduces Bayesian inference via MCMC for neural networks with latent variables, addressing stability and interpretability issues in existing VAEs.

Findings

MCMC faces fundamental challenges in neural networks with latent variables

Imposing parameter constraints improves model stability

Demonstrated on simulated and real datasets

Abstract

In conducting non-linear dimensionality reduction and feature learning, it is common to suppose that the data lie near a lower-dimensional manifold. A class of model-based approaches for such problems includes latent variables in an unknown non-linear regression function; this includes Gaussian process latent variable models and variational auto-encoders (VAEs) as special cases. VAEs are artificial neural networks (ANNs) that employ approximations to make computation tractable; however, current implementations lack adequate uncertainty quantification in estimating the parameters, predictive densities, and lower-dimensional subspace, and can be unstable and lack interpretability in practice. We attempt to solve these problems by deploying Markov chain Monte Carlo sampling algorithms (MCMC) for Bayesian inference in ANN models with latent variables. We address issues of identifiability by imposing constraints on the ANN parameters as well as by using anchor points. This is demonstrated on simulated and real data examples. We find that current MCMC sampling schemes face fundamental challenges in neural networks involving latent variables, motivating new research directions.