Sparse Bayesian neural networks for regression: Tackling overfitting and computational challenges in uncertainty quantification

TL;DR

This paper introduces a sparse Bayesian neural network (SBNN) framework that combines hierarchical Bayesian inference and sparsity priors to improve uncertainty quantification, reduce overfitting, and address computational challenges in regression tasks.

Contribution

The paper proposes a novel SBNN approach using nonlinear sparse Bayesian learning and automatic relevance determination to enhance efficiency and robustness over traditional Bayesian neural networks.

Findings

SBNN effectively prunes redundant parameters, reducing overfitting.

The approach improves uncertainty quantification compared to standard methods.

SBNN demonstrates computational efficiency in regression problems.

Abstract

Neural networks (NNs) are primarily developed within the frequentist statistical framework. Nevertheless, frequentist NNs lack the capability to provide uncertainties in the predictions, and hence their robustness can not be adequately assessed. Conversely, the Bayesian neural networks (BNNs) naturally offer predictive uncertainty by applying Bayes' theorem. However, their computational requirements pose significant challenges. Moreover, both frequentist NNs and BNNs suffer from overfitting issues when dealing with noisy and sparse data, which render their predictions unwieldy away from the available data space. To address both these problems simultaneously, we leverage insights from a hierarchical setting in which the parameter priors are conditional on hyperparameters to construct a BNN by applying a semi-analytical framework known as nonlinear sparse Bayesian learning (NSBL). We call our network sparse Bayesian neural network (SBNN) which aims to address the practical and computational issues associated with BNNs. Simultaneously, imposing a sparsity-inducing prior encourages the automatic pruning of redundant parameters based on the automatic relevance determination (ARD) concept. This process involves removing redundant parameters by optimally selecting the precision of the parameters prior probability density functions (pdfs), resulting in a tractable treatment for overfitting. To demonstrate the benefits of the SBNN algorithm, the study presents an illustrative regression problem and compares the results of a BNN using standard Bayesian inference, hierarchical Bayesian inference, and a BNN equipped with the proposed algorithm. Subsequently, we demonstrate the importance of considering the full parameter posterior by comparing the results with those obtained using the Laplace approximation with and without NSBL.