Calibrated Decomposition of Aleatoric and Epistemic Uncertainty in Deep Features for Inference-Time Adaptation

TL;DR

This paper presents a lightweight framework that disentangles aleatoric and epistemic uncertainties in deep features for more reliable inference-time adaptation, enabling significant computational savings and tighter prediction intervals.

Contribution

It introduces a novel uncertainty decomposition method that requires no sampling or ensembling, improving adaptive model selection and computational efficiency in visual inference.

Findings

Reduces compute by approximately 60% on MOT17 with negligible accuracy loss.

Provides tighter prediction intervals at matched coverage.

Achieves higher computational savings with orthogonal uncertainty decomposition.

Abstract

Most estimators collapse all uncertainty modes into a single confidence score, preventing reliable reasoning about when to allocate more compute or adjust inference. We introduce Uncertainty-Guided Inference-Time Selection, a lightweight inference time framework that disentangles aleatoric (data-driven) and epistemic (model-driven) uncertainty directly in deep feature space. Aleatoric uncertainty is estimated using a regularized global density model, while epistemic uncertainty is formed from three complementary components that capture local support deficiency, manifold spectral collapse, and cross-layer feature inconsistency. These components are empirically orthogonal and require no sampling, no ensembling, and no additional forward passes. We integrate the decomposed uncertainty into a distribution free conformal calibration procedure that yields significantly tighter prediction intervals at matched coverage. Using these components for uncertainty guided adaptive model selection reduces compute by approximately 60 percent on MOT17 with negligible accuracy loss, enabling practical self regulating visual inference. Additionally, our ablation results show that the proposed orthogonal uncertainty decomposition consistently yields higher computational savings across all MOT17 sequences, improving margins by 13.6 percentage points over the total-uncertainty baseline.