Evaluating Prediction Uncertainty Estimates from BatchEnsemble

TL;DR

This paper evaluates BatchEnsemble as a scalable method for uncertainty estimation in deep learning, introducing GRUBE for sequential tasks, and demonstrating its competitive performance against existing methods with efficiency gains.

Contribution

The paper extends BatchEnsemble to sequential models with GRUBE and compares its uncertainty estimation performance to Monte Carlo dropout and deep ensembles.

Findings

BatchEnsemble matches deep ensemble performance in uncertainty estimation.

BatchEnsemble outperforms Monte Carlo dropout.

GRUBE achieves comparable or better results with fewer parameters.

Abstract

Deep learning models struggle with uncertainty estimation. Many approaches are either computationally infeasible or underestimate uncertainty. We investigate \textit{BatchEnsemble} as a general and scalable method for uncertainty estimation across both tabular and time series tasks. To extend BatchEnsemble to sequential modeling, we introduce GRUBE, a novel BatchEnsemble GRU cell. We compare the BatchEnsemble to Monte Carlo dropout and deep ensemble models. Our results show that BatchEnsemble matches the uncertainty estimation performance of deep ensembles, and clearly outperforms Monte Carlo dropout. GRUBE achieves similar or better performance in both prediction and uncertainty estimation. These findings show that BatchEnsemble and GRUBE achieve similar performance with fewer parameters and reduced training and inference time compared to traditional ensembles.