Stochastic Thermodynamics of Learning

TL;DR

This paper applies stochastic thermodynamics to neural network learning, establishing bounds on information acquisition based on thermodynamic costs and analyzing efficiency and optimality conditions.

Contribution

It introduces a thermodynamic framework for understanding neural learning, including efficiency bounds and analysis of Hebbian learning in the thermodynamic limit.

Findings

Information acquired is bounded by thermodynamic cost

Learning efficiency  1

Optimal learning conditions identified

Abstract

Virtually every organism gathers information about its noisy environment and builds models from that data, mostly using neural networks. Here, we use stochastic thermodynamics to analyse the learning of a classification rule by a neural network. We show that the information acquired by the network is bounded by the thermodynamic cost of learning and introduce a learning efficiency $\eta\le1$. We discuss the conditions for optimal learning and analyse Hebbian learning in the thermodynamic limit.