Last Layer Hamiltonian Monte Carlo

TL;DR

This paper introduces Last Layer Hamiltonian Monte Carlo (LL-HMC), a computationally efficient probabilistic method for uncertainty estimation in deep neural networks by sampling only the last layer, demonstrating competitive performance on real-world datasets.

Contribution

The paper proposes LL-HMC, a novel approach that reduces computational costs by applying HMC sampling solely to the last layer of DNNs, enabling scalable uncertainty estimation.

Findings

LL-HMC achieves competitive in-distribution classification.

LL-HMC effectively detects out-of-distribution samples.

Additional sampling does not significantly improve classification performance.

Abstract

We explore the use of Hamiltonian Monte Carlo (HMC) sampling as a probabilistic last layer approach for deep neural networks (DNNs). While HMC is widely regarded as a gold standard for uncertainty estimation, the computational demands limit its application to large-scale datasets and large DNN architectures. Although the predictions from the sampled DNN parameters can be parallelized, the computational cost still scales linearly with the number of samples (similar to an ensemble). Last layer HMC (LL--HMC) reduces the required computations by restricting the HMC sampling to the final layer of a DNN, making it applicable to more data-intensive scenarios with limited computational resources. In this paper, we compare LL-HMC against five last layer probabilistic deep learning (LL-PDL) methods across three real-world video datasets for driver action and intention. We evaluate the in-distribution classification performance, calibration, and out-of-distribution (OOD) detection. Due to the stochastic nature of the probabilistic evaluations, we performed five grid searches for different random seeds to avoid being reliant on a single initialization for the hyperparameter configurations. The results show that LL--HMC achieves competitive in-distribution classification and OOD detection performance. Additional sampled last layer parameters do not improve the classification performance, but can improve the OOD detection. Multiple chains or starting positions did not yield consistent improvements.