Backprop as Functor: A compositional perspective on supervised learning

TL;DR

This paper presents a categorical framework for understanding backpropagation in supervised learning, modeling it as a monoidal functor that offers a structural and generalized perspective on neural networks.

Contribution

It introduces a category-theoretic approach to formalize backpropagation as a functor, broadening the conceptual understanding of neural network training.

Findings

Backpropagation is modeled as a monoidal functor.

Provides a categorical structure for update rules in supervised learning.

Generalizes neural networks through a compositional perspective.

Abstract

A supervised learning algorithm searches over a set of functions $A \to B$ parametrised by a space $P$ to find the best approximation to some ideal function $f\colon A \to B$. It does this by taking examples $(a,f(a)) \in A\times B$, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent---with respect to a fixed step size and an error function satisfying a certain property---defines a monoidal functor from a category of parametrised functions to this category of update rules. This provides a structural perspective on backpropagation, as well as a broad generalisation of neural networks.