Fluctuation-dissipation Type Theorem in Stochastic Linear Learning

TL;DR

This paper derives a generalized fluctuation-dissipation theorem for stochastic linear learning dynamics and validates it across popular machine learning datasets, linking physical principles to learning processes.

Contribution

It introduces a generalized FDT applicable to stochastic linear learning, bridging concepts from physics and machine learning.

Findings

Derived a generalized FDT for stochastic linear learning.

Validated the theorem on MNIST, CIFAR-10, and EMNIST datasets.

Established a link between physical fluctuation-dissipation principles and learning dynamics.

Abstract

The fluctuation-dissipation theorem (FDT) is a simple yet powerful consequence of the first-order differential equation governing the dynamics of systems subject simultaneously to dissipative and stochastic forces. The linear learning dynamics, in which the input vector maps to the output vector by a linear matrix whose elements are the subject of learning, has a stochastic version closely mimicking the Langevin dynamics when a full-batch gradient descent scheme is replaced by that of stochastic gradient descent. We derive a generalized FDT for the stochastic linear learning dynamics and verify its validity among the well-known machine learning data sets such as MNIST, CIFAR-10 and EMNIST.