Statistical Foundation of Variational Bayes Neural Networks

TL;DR

This paper establishes the theoretical validity of variational Bayes methods for neural networks by proving posterior consistency and analyzing convergence rates, thus enhancing their reliability in complex data scenarios.

Contribution

It provides the first theoretical proof of posterior consistency for variational Bayes in neural networks, outlining conditions for convergence and guiding prior construction.

Findings

Proves posterior consistency of variational Bayes in neural networks.

Analyzes the influence of scale parameters on convergence rates.

Provides guidelines for prior distribution design in Bayesian neural networks.

Abstract

Despite the popularism of Bayesian neural networks in recent years, its use is somewhat limited in complex and big data situations due to the computational cost associated with full posterior evaluations. Variational Bayes (VB) provides a useful alternative to circumvent the computational cost and time complexity associated with the generation of samples from the true posterior using Markov Chain Monte Carlo (MCMC) techniques. The efficacy of the VB methods is well established in machine learning literature. However, its potential broader impact is hindered due to a lack of theoretical validity from a statistical perspective. However there are few results which revolve around the theoretical properties of VB, especially in non-parametric problems. In this paper, we establish the fundamental result of posterior consistency for the mean-field variational posterior (VP) for a feed-forward artificial neural network model. The paper underlines the conditions needed to guarantee that the VP concentrates around Hellinger neighborhoods of the true density function. Additionally, the role of the scale parameter and its influence on the convergence rates has also been discussed. The paper mainly relies on two results (1) the rate at which the true posterior grows (2) the rate at which the KL-distance between the posterior and variational posterior grows. The theory provides a guideline of building prior distributions for Bayesian NN models along with an assessment of accuracy of the corresponding VB implementation.