Machines of finite depth: towards a formalization of neural networks

TL;DR

This paper introduces a formal framework called machines of finite depth that unifies neural network architectures, providing modular, efficiently computable, and differentiable models with a unified backpropagation approach.

Contribution

It formalizes neural networks as machines of finite depth, enabling a unified theoretical and practical treatment of various architectures and their training procedures.

Findings

Machines of finite depth are modular and can be combined.

Backpropagation is a machine that can be computed efficiently.

The framework generalizes classical architectures like dense, convolutional, and recurrent networks.

Abstract

We provide a unifying framework where artificial neural networks and their architectures can be formally described as particular cases of a general mathematical construction--machines of finite depth. Unlike neural networks, machines have a precise definition, from which several properties follow naturally. Machines of finite depth are modular (they can be combined), efficiently computable and differentiable. The backward pass of a machine is again a machine and can be computed without overhead using the same procedure as the forward pass. We prove this statement theoretically and practically, via a unified implementation that generalizes several classical architectures--dense, convolutional, and recurrent neural networks with a rich shortcut structure--and their respective backpropagation rules.