The Thermodynamic Costs of Simple Linear Regression

TL;DR

This paper investigates the fundamental thermodynamic costs associated with simple linear regression, deriving bounds and scaling laws that connect energy efficiency with model training and inference.

Contribution

It introduces thermodynamic bounds for linear regression algorithms and proposes energy-aware scaling laws for dataset size and entropy production analysis.

Findings

Derived thermodynamic lower bounds for linear regression

Established energy-cost aware scaling laws for dataset size

Discussed bounds on entropy production from mismatch cost

Abstract

The construction of models from data is a significant contributor to the energetic costs of computation. Because of this, understanding how foundational thermodynamic bounds apply to modeling algorithms will be increasingly important. Here, we study the thermodynamic costs of a basic and fundamental modeling algorithm: simple linear regression. Following Landauer, we approximate the thermodynamic lower bound on irreversibly performing both exact linear regression and linear regression via stochastic gradient descent as implemented on floating-point numbers. From this, we derive energycost aware scaling laws for the optimal dataset size for training a linear regression model given a generalization error dependent demand for inference. Additionally, we discuss a method to lower bound the entropy production from the mismatch cost for algorithms with continuous input variables.