Contrastive learning in tunable dynamical systems

TL;DR

This paper extends contrastive learning theory to dynamical systems governed by differential equations, introducing PAR learning processes to enable scalable training of models with local supervision.

Contribution

It introduces PAR learning processes combining local contrastive learning and scalable supervision for training dynamical system models.

Findings

Local supervision with forward error propagation enables successful training.

Contrastive learning can be generalized beyond equilibrium physical systems.

PAR processes are scalable and effective for biological and machine learning models.

Abstract

We generalize the theory of supervised contrastive learning, previously applied to physical systems at equilibrium or steady state, to systems following any dynamics described by coupled ordinary differential equations. We show that if physical dynamics break time reversal symmetry, gradient descent on a cost function embodying the desired behavior cannot be achieved with a scalable process, even in principle. We therefore introduce Probably Approximately Right (PAR) learning processes, composed of a local contrastive learning rule and a scalable supervision protocol. We show that approximate, local supervision with forward propagation of the error signal can be used to successfully train several tunable models of physical dynamics inspired by examples in biological and machine learning.