Neural Nets via Forward State Transformation and Backward Loss Transformation

TL;DR

This paper presents a semantic perspective on neural networks, viewing forward passes as state transformations and backward passes as predicate transformers, aligning neural network training with program semantics.

Contribution

It introduces a novel semantic framework that interprets neural network transformations as state and predicate transformers, providing a logical foundation for understanding backpropagation.

Findings

Neural networks can be viewed as state transformers in the forward pass.

Backward pass acts as a predicate transformer, changing output losses to input losses.

Backpropagation is shown to be functorial by construction.

Abstract

This article studies (multilayer perceptron) neural networks with an emphasis on the transformations involved --- both forward and backward --- in order to develop a semantical/logical perspective that is in line with standard program semantics. The common two-pass neural network training algorithms make this viewpoint particularly fitting. In the forward direction, neural networks act as state transformers. In the reverse direction, however, neural networks change losses of outputs to losses of inputs, thereby acting like a (real-valued) predicate transformer. In this way, backpropagation is functorial by construction, as shown earlier in recent other work. We illustrate this perspective by training a simple instance of a neural network.