Locally Linear Attributes of ReLU Neural Networks

TL;DR

This paper explores the geometric structure of ReLU neural networks, showing how their piecewise linear nature decomposes input space into convex regions, each with an affine transformation, aiding understanding of their behavior.

Contribution

It provides a detailed analysis of the local linear attributes of ReLU networks, linking weights to input space decomposition and affine mappings.

Findings

Decomposition of input space into convex polytopes

Affine mappings characterize network behavior on each polytope

Structural insights facilitate understanding of neural network functions

Abstract

A ReLU neural network determines/is a continuous piecewise linear map from an input space to an output space. The weights in the neural network determine a decomposition of the input space into convex polytopes and on each of these polytopes the network can be described by a single affine mapping. The structure of the decomposition, together with the affine map attached to each polytope, can be analyzed to investigate the behavior of the associated neural network.