Humble your Overconfident Networks: Unlearning Overfitting via Sequential Monte Carlo Tempered Deep Ensembles

TL;DR

This paper presents a scalable SMC method with SGHMC proposals for Bayesian neural networks, improving uncertainty quantification, calibration, and reducing overfitting in various tasks.

Contribution

Introduces SMCSGHMC, a scalable Bayesian sampling method combining SMC and SGHMC for neural networks, enabling efficient mini-batch sampling and better uncertainty estimates.

Findings

SMCSGHMC outperforms SGD and deep ensembles in image classification.

It reduces overfitting and enhances calibration of neural networks.

The method is effective in OOD detection and transfer learning tasks.

Abstract

Sequential Monte Carlo (SMC) methods offer a principled approach to Bayesian uncertainty quantification but are traditionally limited by the need for full-batch gradient evaluations. We introduce a scalable variant by incorporating Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) proposals into SMC, enabling efficient mini-batch based sampling. Our resulting SMCSGHMC algorithm outperforms standard stochastic gradient descent (SGD) and deep ensembles across image classification, out-of-distribution (OOD) detection, and transfer learning tasks. We further show that SMCSGHMC mitigates overfitting and improves calibration, providing a flexible, scalable pathway for converting pretrained neural networks into well-calibrated Bayesian models.