On the Temperature of Machine Learning Systems

TL;DR

This paper introduces a thermodynamic framework for machine learning systems, defining concepts like temperature and energy to analyze training dynamics and neural network efficiency, bridging physics and ML.

Contribution

It develops a novel thermodynamic theory for ML systems, including defining temperature, states, and phase transitions, and models neural networks as heat engines with efficiency metrics.

Findings

Temperature correlates with data distribution and training complexity.

Neural networks can be classified as heat engines based on work efficiency.

Analytical and asymptotic formulas for system temperature during phase transitions.

Abstract

We develop a thermodynamic theory for machine learning (ML) systems. Similar to physical thermodynamic systems which are characterized by energy and entropy, ML systems possess these characteristics as well. This comparison inspire us to integrate the concept of temperature into ML systems grounded in the fundamental principles of thermodynamics, and establish a basic thermodynamic framework for machine learning systems with non-Boltzmann distributions. We introduce the concept of states within a ML system, identify two typical types of state, and interpret model training and refresh as a process of state phase transition. We consider that the initial potential energy of a ML system is described by the model's loss functions, and the energy adheres to the principle of minimum potential energy. For a variety of energy forms and parameter initialization methods, we derive the temperature of systems during the phase transition both analytically and asymptotically, highlighting temperature as a vital indicator of system data distribution and ML training complexity. Moreover, we perceive deep neural networks as complex heat engines with both global temperature and local temperatures in each layer. The concept of work efficiency is introduced within neural networks, which mainly depends on the neural activation functions. We then classify neural networks based on their work efficiency, and describe neural networks as two types of heat engines.