Minimaxity and Admissibility of Bayesian Neural Networks

TL;DR

This paper investigates the optimality of Bayesian neural networks under decision theory, proposing a hyperprior that achieves minimaxity and admissibility, and validating results through simulations.

Contribution

It introduces a hyperprior on BNN output variance that ensures the resulting decision rule is both minimax and admissible, advancing theoretical understanding of BNN optimality.

Findings

A fixed prior scale leads to non-minimax decision rules.

A hyperprior yields a superharmonic density, ensuring minimaxity and admissibility.

Numerical simulations confirm the theoretical results.

Abstract

Bayesian neural networks (BNNs) offer a natural probabilistic formulation for inference in deep learning models. Despite their popularity, their optimality has received limited attention through the lens of statistical decision theory. In this paper, we study decision rules induced by deep, fully connected feedforward ReLU BNNs in the normal location model under quadratic loss. We show that, for fixed prior scales, the induced Bayes decision rule is not minimax. We then propose a hyperprior on the effective output variance of the BNN prior that yields a superharmonic square-root marginal density, establishing that the resulting decision rule is simultaneously admissible and minimax. We further extend these results from the quadratic loss setting to the predictive density estimation problem with Kullback--Leibler loss. Finally, we validate our theoretical findings numerically through simulation.