Learning with springs and sticks

TL;DR

This paper introduces a physical dynamical system of springs and sticks that can approximate any continuous function, using energy minimization and thermodynamic principles to understand learning processes.

Contribution

It presents a novel physical model for learning that combines mechanical components with energy-based optimization and explores its thermodynamic properties.

Findings

System can perform regression with performance comparable to neural networks.

Identifies a thermodynamic learning barrier caused by environmental fluctuations.

Links free energy changes to the system's ability to learn data distributions.

Abstract

Learning is a physical process. Here, we aim to study a simple dynamical system composed of springs and sticks capable of arbitrarily approximating any continuous function. The main idea of our work is to use the sticks to mimic a piecewise-linear approximation of the given function, use the potential energy of springs to encode a desired mean squared error loss function, and converge to a minimum-energy configuration via dissipation. We apply the proposed simulation system to regression tasks and show that its performance is comparable to that of multi-layer perceptrons. In addition, we study the thermodynamic properties of the system and find a relation between the free energy change of the system and its ability to learn an underlying data distribution. We empirically find a \emph{thermodynamic learning barrier} for the system caused by the fluctuations of the environment, whereby the system cannot learn if its change in free energy hits such a barrier. We believe this simple model can help us better understand learning systems from a physical point of view.