Hyperparameter Ensembles for Robustness and Uncertainty Quantification

TL;DR

This paper introduces hyperparameter ensembles, including hyper-deep and hyper-batch ensembles, which combine weight and hyperparameter diversity to improve accuracy, calibration, and efficiency in neural network ensemble methods.

Contribution

The paper proposes novel hyperparameter ensemble methods that outperform existing deep and batch ensembles in accuracy and efficiency across multiple neural network architectures.

Findings

Hyper-deep ensembles outperform traditional ensembles in accuracy.

Hyper-batch ensembles reduce computational and memory costs.

Combining weight and hyperparameter diversity enhances model robustness.

Abstract

Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration. The recently introduced batch ensembles provide a drop-in replacement that is more parameter efficient. In this paper, we design ensembles not only over weights, but over hyperparameters to improve the state of the art in both settings. For best performance independent of budget, we propose hyper-deep ensembles, a simple procedure that involves a random search over different hyperparameters, themselves stratified across multiple random initializations. Its strong performance highlights the benefit of combining models with both weight and hyperparameter diversity. We further propose a parameter efficient version, hyper-batch ensembles, which builds on the layer structure of batch ensembles and self-tuning networks. The computational and memory costs of our method are notably lower than typical ensembles. On image classification tasks, with MLP, LeNet, ResNet 20 and Wide ResNet 28-10 architectures, we improve upon both deep and batch ensembles.