Nonequilibrium thermodynamics of self-supervised learning

TL;DR

This paper models self-supervised learning as a nonequilibrium thermodynamic process, revealing how it operates through thermodynamic cycles and feedback mechanisms, and introduces a generalized Gibbs ensemble perspective.

Contribution

It introduces a novel thermodynamic framework for SSL, connecting it to nonequilibrium thermodynamics and generalized Gibbs ensembles, and interprets learning as a feedback-driven thermodynamic cycle.

Findings

SSL systems can be modeled as thermodynamic cycles.

Learning operates as a feedback cycle extracting negative work.

SSL algorithms can be understood through thermodynamic principles.

Abstract

Self-supervised learning (SSL) of energy based models has an intuitive relation to equilibrium thermodynamics because the softmax layer, mapping energies to probabilities, is a Gibbs distribution. However, in what way SSL is a thermodynamic process? We show that some SSL paradigms behave as a thermodynamic composite system formed by representations and self-labels in contact with a nonequilibrium reservoir. Moreover, this system is subjected to usual thermodynamic cycles, such as adiabatic expansion and isochoric heating, resulting in a generalized Gibbs ensemble (GGE). In this picture, we show that learning is seen as a demon that operates in cycles using feedback measurements to extract negative work from the system. As applications, we examine some SSL algorithms using this idea.