PDE constraints on smooth hierarchical functions computed by neural networks

TL;DR

This paper explores the algebraic PDE constraints characterizing smooth hierarchical functions computed by neural networks, revealing how network topology influences the functions' differential properties and offering insights into their expressivity.

Contribution

It establishes PDE constraints dependent only on network topology for smooth hierarchical functions, advancing the algebraic understanding of neural network expressivity.

Findings

Existence of PDEs satisfied by functions computed by neural networks.

PDEs depend solely on network topology and are algebraic in nature.

Conjecture that PDE constraints characterize network-representable functions.

Abstract

Neural networks are versatile tools for computation, having the ability to approximate a broad range of functions. An important problem in the theory of deep neural networks is expressivity; that is, we want to understand the functions that are computable by a given network. We study real infinitely differentiable (smooth) hierarchical functions implemented by feedforward neural networks via composing simpler functions in two cases: 1) each constituent function of the composition has fewer inputs than the resulting function; 2) constituent functions are in the more specific yet prevalent form of a non-linear univariate function (e.g. tanh) applied to a linear multivariate function. We establish that in each of these regimes there exist non-trivial algebraic partial differential equations (PDEs), which are satisfied by the computed functions. These PDEs are purely in terms of the partial derivatives and are dependent only on the topology of the network. For compositions of polynomial functions, the algebraic PDEs yield non-trivial equations (of degrees dependent only on the architecture) in the ambient polynomial space that are satisfied on the associated functional varieties. Conversely, we conjecture that such PDE constraints, once accompanied by appropriate non-singularity conditions and perhaps certain inequalities involving partial derivatives, guarantee that the smooth function under consideration can be represented by the network. The conjecture is verified in numerous examples including the case of tree architectures which are of neuroscientific interest. Our approach is a step toward formulating an algebraic description of functional spaces associated with specific neural networks, and may provide new, useful tools for constructing neural networks.