Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimation

TL;DR

This paper introduces Variance-Gated Ensembles (VGE), a novel, scalable framework for more reliable epistemic uncertainty estimation in ensemble models, addressing limitations of traditional additive decomposition methods.

Contribution

VGE provides a differentiable, end-to-end trainable framework that enhances epistemic uncertainty estimation by integrating ensemble statistics through a variance-gated mechanism.

Findings

VGE matches or exceeds state-of-the-art uncertainty baselines.

VGE is computationally efficient and scalable.

VGE enables end-to-end training with closed-form vector-Jacobian products.

Abstract

Machine learning applications require fast and reliable per-sample uncertainty estimation. A common approach is to use predictive distributions from Bayesian or approximation methods and additively decompose uncertainty into aleatoric (i.e., data-related) and epistemic (i.e., model-related) components. However, additive decomposition has recently been questioned, with evidence that it breaks down when using finite-ensemble sampling and/or mismatched predictive distributions. This paper introduces Variance-Gated Ensembles (VGE), an intuitive, differentiable framework that injects epistemic sensitivity via a signal-to-noise gate computed from ensemble statistics. VGE provides: (i) a Variance-Gated Margin Uncertainty (VGMU) score that couples decision margins with ensemble predictive variance; and (ii) a Variance-Gated Normalization (VGN) layer that generalizes the variance-gated uncertainty mechanism to training via per-class, learnable normalization of ensemble member probabilities. We derive closed-form vector-Jacobian products enabling end-to-end training through ensemble sample mean and variance. VGE matches or exceeds state-of-the-art information-theoretic baselines while remaining computationally efficient. As a result, VGE provides a practical and scalable approach to epistemic-aware uncertainty estimation in ensemble models. An open-source implementation is available at: https://github.com/nextdevai/vge.