On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks

TL;DR

This paper investigates how the architecture of ReLU neural networks influences the topology of their decision regions, establishing conditions under which these regions have limited bounded components.

Contribution

It introduces the concept of generic, transversal ReLU networks and proves bounds on the number of bounded connected components of decision regions based on network architecture.

Findings

Almost all ReLU networks are generic and transversal.

Decision regions of certain ReLU networks can have at most one bounded component.

Provides topological constraints on neural network decision boundaries.

Abstract

Let F:R^n -> R be a feedforward ReLU neural network. It is well-known that for any choice of parameters, F is continuous and piecewise (affine) linear. We lay some foundations for a systematic investigation of how the architecture of F impacts the geometry and topology of its possible decision regions for binary classification tasks. Following the classical progression for smooth functions in differential topology, we first define the notion of a generic, transversal ReLU neural network and show that almost all ReLU networks are generic and transversal. We then define a partially-oriented linear 1-complex in the domain of F and identify properties of this complex that yield an obstruction to the existence of bounded connected components of a decision region. We use this obstruction to prove that a decision region of a generic, transversal ReLU network F: R^n -> R with a single hidden layer of dimension (n + 1) can have no more than one bounded connected component.