The Geometry of ReLU Networks through the ReLU Transition Graph

TL;DR

This paper introduces the ReLU Transition Graph (RTG), a combinatorial framework that models the linear regions of ReLU networks to analyze their expressivity, generalization, and robustness through graph-theoretic properties.

Contribution

It develops a novel graph-based theoretical framework for ReLU networks, providing bounds, connectivity proofs, and interpretations of network complexity measures.

Findings

RTG size and diameter bounds established

RTG connectivity proven

Graph properties linked to generalization error

Abstract

We develop a novel theoretical framework for analyzing ReLU neural networks through the lens of a combinatorial object we term the ReLU Transition Graph (RTG). In this graph, each node corresponds to a linear region induced by the network's activation patterns, and edges connect regions that differ by a single neuron flip. Building on this structure, we derive a suite of new theoretical results connecting RTG geometry to expressivity, generalization, and robustness. Our contributions include tight combinatorial bounds on RTG size and diameter, a proof of RTG connectivity, and graph-theoretic interpretations of VC-dimension. We also relate entropy and average degree of the RTG to generalization error. Each theoretical result is rigorously validated via carefully controlled experiments across varied network depths, widths, and data regimes. This work provides the first unified treatment of ReLU network structure via graph theory and opens new avenues for compression, regularization, and complexity control rooted in RTG analysis.