Activation-Space Uncertainty Quantification for Pretrained Networks

TL;DR

This paper introduces GAPA, a post-hoc method that quantifies uncertainty in pretrained networks by shifting Bayesian modeling to activation space, enabling efficient and accurate uncertainty estimates without retraining or sampling.

Contribution

GAPA is a novel approach that replaces nonlinearities with Gaussian-process activations, preserving predictions and providing closed-form uncertainty estimates in a scalable manner.

Findings

GAPA matches or outperforms strong baselines in calibration.

GAPA improves out-of-distribution detection.

GAPA operates efficiently at test time without sampling.

Abstract

Reliable uncertainty estimates are crucial for deploying pretrained models; yet, many strong methods for quantifying uncertainty require retraining, Monte Carlo sampling, or expensive second-order computations and may alter a frozen backbone's predictions. To address this, we introduce Gaussian Process Activations (GAPA), a post-hoc method that shifts Bayesian modeling from weights to activations. GAPA replaces standard nonlinearities with Gaussian-process activations whose posterior mean exactly matches the original activation, preserving the backbone's point predictions by construction while providing closed-form epistemic variances in activation space. To scale to modern architectures, we use a sparse variational inducing-point approximation over cached training activations, combined with local k-nearest-neighbor subset conditioning, enabling deterministic single-pass uncertainty propagation without sampling, backpropagation, or second-order information. Across regression, classification, image segmentation, and language modeling, GAPA matches or outperforms strong post-hoc baselines in calibration and out-of-distribution detection while remaining efficient at test time.