Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning

TL;DR

This paper introduces Fixed-Mean Gaussian Processes (FMGP), a scalable method that enhances uncertainty estimation in pre-trained deep neural networks by fixing the GP mean to the network's outputs, improving both accuracy and efficiency.

Contribution

The paper proposes a novel fixed-mean Gaussian process framework that leverages pre-trained DNN outputs for improved uncertainty estimation without retraining the entire model.

Findings

FMGP improves uncertainty calibration over existing methods.

The approach scales efficiently to large datasets like ImageNet.

FMGP enhances both uncertainty estimation and computational efficiency.

Abstract

Recently, there has been an increasing interest in performing post-hoc uncertainty estimation about the predictions of pre-trained deep neural networks (DNNs). Given a pre-trained DNN via back-propagation, these methods enhance the original network by adding output confidence measures, such as error bars, without compromising its initial accuracy. In this context, we introduce a novel family of sparse variational Gaussian processes (GPs), where the posterior mean is fixed to any continuous function when using a universal kernel. Specifically, we fix the mean of this GP to the output of the pre-trained DNN, allowing our approach to effectively fit the GP's predictive variances to estimate the DNN prediction uncertainty. Our approach leverages variational inference (VI) for efficient stochastic optimization, with training costs that remain independent of the number of training points, scaling efficiently to large datasets such as ImageNet. The proposed method, called fixed mean GP (FMGP), is architecture-agnostic, relying solely on the pre-trained model's outputs to adjust the predictive variances. Experimental results demonstrate that FMGP improves both uncertainty estimation and computational efficiency when compared to state-of-the-art methods.