Neural network layers as parametric spans

TL;DR

This paper introduces a categorical framework for neural network layers using parametric spans, providing a unified mathematical foundation that generalizes classical layers and ensures differentiability for backpropagation.

Contribution

It presents a novel, general definition of neural network layers based on integration theory and parametric spans, unifying classical layers within a rigorous mathematical framework.

Findings

Generalized layer definition encompasses dense and convolutional layers.

Guarantees existence and computability of derivatives for backpropagation.

Provides a mathematically rigorous foundation for neural network layer design.

Abstract

Properties such as composability and automatic differentiation made artificial neural networks a pervasive tool in applications. Tackling more challenging problems caused neural networks to progressively become more complex and thus difficult to define from a mathematical perspective. We present a general definition of linear layer arising from a categorical framework based on the notions of integration theory and parametric spans. This definition generalizes and encompasses classical layers (e.g., dense, convolutional), while guaranteeing existence and computability of the layer's derivatives for backpropagation.