Functional dimension of feedforward ReLU neural networks

TL;DR

This paper investigates the concept of functional dimension in ReLU neural networks, revealing its variability across parameter space and analyzing the structure of fibers and symmetries in the function realization map.

Contribution

It introduces the notion of functional dimension, demonstrates its inhomogeneity, and explores the structure of fibers and symmetries in ReLU neural networks.

Findings

Functional dimension varies across parameter space.

Fibers can be disconnected and have non-constant functional dimension.

Symmetry groups may act non-transitively on fibers.

Abstract

It is well-known that the parameterized family of functions representable by fully-connected feedforward neural networks with ReLU activation function is precisely the class of piecewise linear functions with finitely many pieces. It is less well-known that for every fixed architecture of ReLU neural network, the parameter space admits positive-dimensional spaces of symmetries, and hence the local functional dimension near any given parameter is lower than the parametric dimension. In this work we carefully define the notion of functional dimension, show that it is inhomogeneous across the parameter space of ReLU neural network functions, and continue an investigation - initiated in [14] and [5] - into when the functional dimension achieves its theoretical maximum. We also study the quotient space and fibers of the realization map from parameter space to function space, supplying examples of fibers that are disconnected, fibers upon which functional dimension is non-constant, and fibers upon which the symmetry group acts non-transitively.