Online Laplace Model Selection Revisited

TL;DR

This paper revisits online Laplace model selection for neural networks, providing a new derivation that addresses previous assumptions and demonstrating practical benefits such as preventing overfitting.

Contribution

It re-derives online Laplace methods to target a mode-corrected variational bound, clarifying their theoretical foundation and practical effectiveness.

Findings

Online algorithms attain stationary points with maximum a posteriori parameters.

Optimized hyperparameters prevent overfitting.

Outperform validation-based early stopping on UCI datasets.

Abstract

The Laplace approximation provides a closed-form model selection objective for neural networks (NN). Online variants, which optimise NN parameters jointly with hyperparameters, like weight decay strength, have seen renewed interest in the Bayesian deep learning community. However, these methods violate Laplace's method's critical assumption that the approximation is performed around a mode of the loss, calling into question their soundness. This work re-derives online Laplace methods, showing them to target a variational bound on a mode-corrected variant of the Laplace evidence which does not make stationarity assumptions. Online Laplace and its mode-corrected counterpart share stationary points where 1. the NN parameters are a maximum a posteriori, satisfying the Laplace method's assumption, and 2. the hyperparameters maximise the Laplace evidence, motivating online methods. We demonstrate that these optima are roughly attained in practise by online algorithms using full-batch gradient descent on UCI regression datasets. The optimised hyperparameters prevent overfitting and outperform validation-based early stopping.