Constraining the outputs of ReLU neural networks

TL;DR

This paper introduces algebraic varieties associated with ReLU neural networks, characterizing their outputs through polynomial equations derived from the networks' piecewise linear structure, providing insights into their expressive capabilities.

Contribution

It develops a novel algebraic framework for analyzing ReLU networks using polynomial equations and studies conditions for these varieties to reach expected dimensions.

Findings

Polynomial equations characterize network outputs within activation regions.

Conditions identified for varieties to attain expected dimension.

Provides structural insights into the expressive power of ReLU networks.

Abstract

We introduce a class of algebraic varieties naturally associated with ReLU neural networks, arising from the piecewise linear structure of their outputs across activation regions in input space, and the piecewise multilinear structure in parameter space. By analyzing the rank constraints on the network outputs within each activation region, we derive polynomial equations that characterize the functions representable by the network. We further investigate conditions under which these varieties attain their expected dimension, providing insight into the expressive and structural properties of ReLU networks.