From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network

TL;DR

This paper introduces the Hierarchical Approximate Bayesian Neural Network (HABNN), which enhances neural network robustness and uncertainty estimation by using a Gaussian-inverse-Wishart hyperprior, demonstrating superior performance especially on out-of-distribution data.

Contribution

The paper proposes a novel hierarchical Bayesian neural network model with a Gaussian-inverse-Wishart hyperprior, providing analytical solutions for predictive distribution and weight posterior in closed form.

Findings

HABNN effectively mitigates overfitting.

It provides reliable uncertainty estimates for out-of-distribution data.

The model often outperforms state-of-the-art approaches.

Abstract

In recent years, neural networks have revolutionized various domains, yet challenges such as hyperparameter tuning and overfitting remain significant hurdles. Bayesian neural networks offer a framework to address these challenges by incorporating uncertainty directly into the model, yielding more reliable predictions, particularly for out-of-distribution data. This paper presents Hierarchical Approximate Bayesian Neural Network, a novel approach that uses a Gaussian-inverse-Wishart distribution as a hyperprior of the network's weights to increase both the robustness and performance of the model. We provide analytical representations for the predictive distribution and weight posterior, which amount to the calculation of the parameters of Student's t-distributions in closed form with linear complexity with respect to the number of weights. Our method demonstrates robust performance, effectively addressing issues of overfitting and providing reliable uncertainty estimates, particularly for out-of-distribution tasks. Experimental results indicate that HABNN not only matches but often outperforms state-of-the-art models, suggesting a promising direction for future applications in safety-critical environments.