Neural Bootstrapper

TL;DR

Neural Bootstrapper (NeuBoots) offers a computationally efficient way to generate bootstrapped neural networks by learning to produce bootstrap samples in a single training run, improving uncertainty estimation tasks.

Contribution

NeuBoots introduces a novel method that learns to generate bootstrap samples within a single neural network training, reducing computational costs significantly.

Findings

NeuBoots outperforms traditional bagging methods in accuracy.

NeuBoots achieves lower computational costs while maintaining bootstrap validity.

NeuBoots improves uncertainty quantification in various tasks.

Abstract

Bootstrapping has been a primary tool for ensemble and uncertainty quantification in machine learning and statistics. However, due to its nature of multiple training and resampling, bootstrapping deep neural networks is computationally burdensome; hence it has difficulties in practical application to the uncertainty estimation and related tasks. To overcome this computational bottleneck, we propose a novel approach called \emph{Neural Bootstrapper} (NeuBoots), which learns to generate bootstrapped neural networks through single model training. NeuBoots injects the bootstrap weights into the high-level feature layers of the backbone network and outputs the bootstrapped predictions of the target, without additional parameters and the repetitive computations from scratch. We apply NeuBoots to various machine learning tasks related to uncertainty quantification, including prediction calibrations in image classification and semantic segmentation, active learning, and detection of out-of-distribution samples. Our empirical results show that NeuBoots outperforms other bagging based methods under a much lower computational cost without losing the validity of bootstrapping.