A Unified Representation of Neural Networks Architectures

TL;DR

This paper develops a unified, deterministic framework called DiPaNet that models neural network architectures as limits of infinitely wide and deep networks, connecting various existing models.

Contribution

It introduces DiPaNet, a homogeneous representation linking finite and infinite neural networks through homogenization and discretization techniques.

Findings

Derives approximation errors for infinite-width neural networks.

Formalizes the relation between neural ODEs and residual networks.

Unifies various neural network architectures within a single framework.

Abstract

In this paper we consider the limiting case of neural networks (NNs) architectures when the number of neurons in each hidden layer and the number of hidden layers tend to infinity thus forming a continuum, and we derive approximation errors as a function of the number of neurons and/or hidden layers. Firstly, we consider the case of neural networks with a single hidden layer and we derive an integral infinite width neural representation that generalizes existing continuous neural networks (CNNs) representations. Then we extend this to deep residual CNNs that have a finite number of integral hidden layers and residual connections. Secondly, we revisit the relation between neural ODEs and deep residual NNs and we formalize approximation errors via discretization techniques. Then, we merge these two approaches into a unified homogeneous representation of NNs as a Distributed Parameter neural Network (DiPaNet) and we show that most of the existing finite and infinite-dimensional NNs architectures are related via homogenization/discretization with the DiPaNet representation. Our approach is purely deterministic and applies to general, uniformly continuous matrix weight functions. Relations with neural fields and other neural integro-differential equations are discussed along with further possible generalizations and applications of the DiPaNet framework.