# Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles

## TL;DR

This paper introduces Maximize Overall Diversity (MOD), a simple method to enhance uncertainty estimates in neural network ensembles by increasing their diversity, leading to better out-of-distribution predictions and Bayesian optimization performance.

## Contribution

MOD is a novel approach that encourages larger overall diversity in ensemble predictions, significantly improving uncertainty estimation without harming in-distribution accuracy.

## Key findings

- Improves out-of-distribution prediction accuracy across multiple datasets.
- Enhances Bayesian optimization performance using MOD-based uncertainty estimates.
- Does not compromise in-distribution performance.

## Abstract

The inaccuracy of neural network models on inputs that do not stem from the training data distribution is both problematic and at times unrecognized. Model uncertainty estimation can address this issue, where uncertainty estimates are often based on the variation in predictions produced by a diverse ensemble of models applied to the same input. Here we describe Maximize Overall Diversity (MOD), a straightforward approach to improve ensemble-based uncertainty estimates by encouraging larger overall diversity in ensemble predictions across all possible inputs that might be encountered in the future. When applied to various neural network ensembles, MOD significantly improves predictive performance for out-of-distribution test examples without sacrificing in-distribution performance on 38 Protein-DNA binding regression datasets, 9 UCI datasets, and the IMDB-Wiki image dataset. Across many Bayesian optimization tasks, the performance of UCB acquisition is also greatly improved by leveraging MOD uncertainty estimates.