Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network

TL;DR

This paper introduces a parameter-efficient ensemble patching method for variational Bayesian neural networks that overcomes high parameter and implementation overhead, improving accuracy and calibration on large-scale image classification.

Contribution

It proposes a new variational family for ensemble Bayesian neural networks that works well with batch normalization, reducing overhead and enhancing performance.

Findings

Improved predictive accuracy on ImageNet with ResNet-18.

Achieved near-perfect calibration in Bayesian neural networks.

Reduced parameter and programming overhead compared to traditional methods.

Abstract

Two main obstacles preventing the widespread adoption of variational Bayesian neural networks are the high parameter overhead that makes them infeasible on large networks, and the difficulty of implementation, which can be thought of as "programming overhead." MC dropout [Gal and Ghahramani, 2016] is popular because it sidesteps these obstacles. Nevertheless, dropout is often harmful to model performance when used in networks with batch normalization layers [Li et al., 2018], which are an indispensable part of modern neural networks. We construct a general variational family for ensemble-based Bayesian neural networks that encompasses dropout as a special case. We further present two specific members of this family that work well with batch normalization layers, while retaining the benefits of low parameter and programming overhead, comparable to non-Bayesian training. Our proposed methods improve predictive accuracy and achieve almost perfect calibration on a ResNet-18 trained with ImageNet.