The Geometric Structure of Fully-Connected ReLU Layers

TL;DR

This paper explores the geometric structure of fully-connected ReLU neural network layers, revealing how they partition input space and simplify analysis of decision boundaries, with implications for understanding network complexity.

Contribution

It formalizes the geometric interpretation of ReLU layers as projections onto polyhedral cones and analyzes the complexity of decision boundaries in shallow networks.

Findings

ReLU layers induce a natural partition of input space.

A single hidden ReLU layer can generate at most d different decision boundaries.

Adding more layers affects the geometric complexity of the network.

Abstract

We formalize and interpret the geometric structure of $d$-dimensional fully connected ReLU layers in neural networks. The parameters of a ReLU layer induce a natural partition of the input domain, such that the ReLU layer can be significantly simplified in each sector of the partition. This leads to a geometric interpretation of a ReLU layer as a projection onto a polyhedral cone followed by an affine transformation, in line with the description in [doi:10.48550/arXiv.1905.08922] for convolutional networks with ReLU activations. Further, this structure facilitates simplified expressions for preimages of the intersection between partition sectors and hyperplanes, which is useful when describing decision boundaries in a classification setting. We investigate this in detail for a feed-forward network with one hidden ReLU-layer, where we provide results on the geometric complexity of the decision boundary generated by such networks, as well as proving that modulo an affine transformation, such a network can only generate $d$ different decision boundaries. Finally, the effect of adding more layers to the network is discussed.