On the Local Complexity of Linear Regions in Deep ReLU Networks

TL;DR

This paper introduces a theoretical measure called local complexity to analyze how ReLU neural networks learn features and how their geometric properties relate to robustness and function variation.

Contribution

It provides a new framework linking the local complexity of ReLU networks to feature learning, robustness, and the effects of optimization on network solutions.

Findings

ReLU networks with low-dimensional features have lower local complexity.

Local complexity bounds the total variation of the learned function.

Feature learning relates to increased adversarial robustness.

Abstract

We define the local complexity of a neural network with continuous piecewise linear activations as a measure of the density of linear regions over an input data distribution. We show theoretically that ReLU networks that learn low-dimensional feature representations have a lower local complexity. This allows us to connect recent empirical observations on feature learning at the level of the weight matrices with concrete properties of the learned functions. In particular, we show that the local complexity serves as an upper bound on the total variation of the function over the input data distribution and thus that feature learning can be related to adversarial robustness. Lastly, we consider how optimization drives ReLU networks towards solutions with lower local complexity. Overall, this work contributes a theoretical framework towards relating geometric properties of ReLU networks to different aspects of learning such as feature learning and representation cost.