Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks

TL;DR

This paper introduces a framework using activation histograms to bound the number of affine regions in ReLU neural networks, providing tighter bounds and insights into parameter initialization effects.

Contribution

It extends existing bounds by analyzing algebraic topology aspects and proposes a method to compose subnetworks for improved bounds.

Findings

Tighter bounds on affine regions are achieved.

Parameter initialization influences the number of regions.

Subnetwork composition reduces the number of necessary bounds.

Abstract

Several current bounds on the maximal number of affine regions of a ReLU feed-forward neural network are special cases of the framework [1] which relies on layer-wise activation histogram bounds. We analyze and partially solve a problem in algebraic topology the solution of which would fully exploit this framework. Our partial solution already induces slightly tighter bounds and suggests insight in how parameter initialization methods can affect the number of regions. Furthermore, we extend the framework to allow the composition of subnetwork instead of layer-wise activation histogram bounds to reduce the number of required compositions which negatively affect the tightness of the resulting bound.