Topological properties of the set of functions generated by neural networks of fixed size

TL;DR

This paper investigates the topological structure of the set of functions representable by fixed-size neural networks, revealing non-convexity, non-closure, and instability issues that may hinder training convergence.

Contribution

It provides a detailed analysis of the topological limitations of neural network function spaces, highlighting properties that could cause training difficulties.

Findings

Set is highly non-convex for most activation functions.

Set is not closed in various norms, except for ReLU and parametric ReLU.

Function-to-weights mapping is not inverse stable, affecting training stability.

Abstract

We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties. It is highly non-convex, except possibly for a few exotic activation functions. Moreover, the set is not closed with respect to $L^p$-norms, $0 < p < \infty$, for all practically-used activation functions, and also not closed with respect to the $L^\infty$-norm for all practically-used activation functions except for the ReLU and the parametric ReLU. Finally, the function that maps a family of weights to the function computed by the associated network is not inverse stable for every practically used activation function. In other words, if $f_1, f_2$ are two functions realized by neural networks and if $f_1, f_2$ are close in the sense that $\|f_1 - f_2\|_{L^\infty} \leq \varepsilon$ for $\varepsilon > 0$, it is, regardless of the size of $\varepsilon$, usually not possible to find weights $w_1, w_2$ close together such that each $f_i$ is realized by a neural network with weights $w_i$. Overall, our findings identify potential causes for issues in the training procedure of deep learning such as no guaranteed convergence, explosion of parameters, and slow convergence.