Fluctuation-dissipation relations for stochastic gradient descent

TL;DR

This paper derives fluctuation-dissipation relations for stochastic gradient descent that connect measurable quantities to hyperparameters, enabling adaptive training and insights into the loss landscape, with empirical validation.

Contribution

It introduces exact fluctuation-dissipation relations for SGD's stationary states, facilitating adaptive training and landscape analysis.

Findings

Relations hold exactly for any stationary state.

Can be used to adaptively set training schedules.

Efficiently extract Hessian and landscape information.

Abstract

The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm. These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule. We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity. Our claims are empirically verified.