Probabilistic Skip Connections for Deterministic Uncertainty Quantification in Deep Neural Networks

TL;DR

This paper introduces probabilistic skip connections (PSCs) that leverage neural collapse measures to enable effective uncertainty quantification and out-of-distribution detection in deep neural networks without retraining.

Contribution

The authors propose using neural collapse measures to identify suitable layers for probabilistic modeling, allowing retrofitting existing models for uncertainty quantification without retraining.

Findings

PSCs effectively disentangle aleatoric and epistemic uncertainty.

PSCs match or outperform existing methods in OOD detection.

Spectral normalization influences neural collapse and PSC effectiveness.

Abstract

Deterministic uncertainty quantification (UQ) in deep learning aims to estimate uncertainty with a single pass through a network by leveraging outputs from the network's feature extractor. Existing methods require that the feature extractor be both sensitive and smooth, ensuring meaningful input changes produce meaningful changes in feature vectors. Smoothness enables generalization, while sensitivity prevents feature collapse, where distinct inputs are mapped to identical feature vectors. To meet these requirements, current deterministic methods often retrain networks with spectral normalization. Instead of modifying training, we propose using measures of neural collapse to identify an existing intermediate layer that is both sensitive and smooth. We then fit a probabilistic model to the feature vector of this intermediate layer, which we call a probabilistic skip connection (PSC). Through empirical analysis, we explore the impact of spectral normalization on neural collapse and demonstrate that PSCs can effectively disentangle aleatoric and epistemic uncertainty. Additionally, we show that PSCs achieve uncertainty quantification and out-of-distribution (OOD) detection performance that matches or exceeds existing single-pass methods requiring training modifications. By retrofitting existing models, PSCs enable high-quality UQ and OOD capabilities without retraining.