Twin-Boot: Uncertainty-Aware Optimization via Online Two-Sample Bootstrapping

TL;DR

Twin-Boot introduces a novel training method that integrates uncertainty estimation directly into the optimization process of deep neural networks, improving calibration, generalization, and interpretability in complex tasks.

Contribution

It presents Twin-Boot, a resampling-based optimization method that trains two models in parallel on bootstrap samples to estimate and utilize uncertainty during training.

Findings

Improves model calibration and generalization in deep learning.

Provides interpretable uncertainty maps in high-dimensional problems.

Enhances robustness by regularizing towards flatter minima.

Abstract

Standard gradient descent methods yield point estimates with no measure of confidence. This limitation is acute in overparameterized and low-data regimes, where models have many parameters relative to available data and can easily overfit. Bootstrapping is a classical statistical framework for uncertainty estimation based on resampling, but naively applying it to deep learning is impractical: it requires training many replicas, produces post-hoc estimates that cannot guide learning, and implicitly assumes comparable optima across runs - an assumption that fails in non-convex landscapes. We introduce Twin-Bootstrap Gradient Descent (Twin-Boot), a resampling-based training procedure that integrates uncertainty estimation into optimization. Two identical models are trained in parallel on independent bootstrap samples, and a periodic mean-reset keeps both trajectories in the same basin so that their divergence reflects local (within-basin) uncertainty. During training, we use this estimate to sample weights in an adaptive, data-driven way, providing regularization that favors flatter solutions. In deep neural networks and complex high-dimensional inverse problems, the approach improves calibration and generalization and yields interpretable uncertainty maps.