On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks

TL;DR

This paper investigates whether exploring local regions around solutions in Bayesian neural networks improves predictive calibration and accuracy, finding that simple mode exploration methods offer limited benefits over standard ensembles.

Contribution

It evaluates the utility of local mode exploration techniques in Bayesian neural networks, revealing minimal improvements over traditional deep ensemble methods.

Findings

Adding modes improves performance slightly

Simple mode exploration methods yield limited gains

Ensemble diversity remains crucial for calibration

Abstract

Multiple techniques for producing calibrated predictive probabilities using deep neural networks in supervised learning settings have emerged that leverage approaches to ensemble diverse solutions discovered during cyclic training or training from multiple random starting points (deep ensembles). However, only a limited amount of work has investigated the utility of exploring the local region around each diverse solution (posterior mode). Using three well-known deep architectures on the CIFAR-10 dataset, we evaluate several simple methods for exploring local regions of the weight space with respect to Brier score, accuracy, and expected calibration error. We consider both Bayesian inference techniques (variational inference and Hamiltonian Monte Carlo applied to the softmax output layer) as well as utilizing the stochastic gradient descent trajectory near optima. While adding separate modes to the ensemble uniformly improves performance, we show that the simple mode exploration methods considered here produce little to no improvement over ensembles without mode exploration.