Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics

TL;DR

The paper introduces DiMS, a novel sampling method based on dissipative Riemannian mechanics, for accurately exploring connected minima in neural network loss landscapes, enhancing Bayesian uncertainty quantification.

Contribution

It proposes a physically inspired dynamical system for sampling reparameterization invariant solutions, guaranteeing exact sampling from minimum level sets.

Findings

DiMS improves exploration of loss minima regions.

Enhanced Bayesian uncertainty quantification performance.

Guarantees exact sampling from invariant solution sets.

Abstract

The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data. Analytically characterizing these solutions is a hard problem, but sampling approaches are feasible. By construction, existing methods either spread over low-loss regions, and thus do not sample reparameterization invariant solutions exactly, or are inherently local, which limits exploration of other minima valleys. We propose sampling such reparameterization invariant models using a dynamical system based on kinetic energy, subject to a gravitational pull and a friction term that dissipates energy from the system. Our proposed sampler, DiMS, is guaranteed to sample exactly from the minimum level sets and depends on physically motivated hyperparameters which allows control over the exploration capabilities of the sampler. We consider uncertainty quantification in Bayesian inference as the motivating problem and observe improved performance compared to previously proposed approaches.