A General Computational Framework to Measure the Expressiveness of Complex Networks Using a Tighter Upper Bound of Linear Regions

TL;DR

This paper introduces a new, tighter upper bound for the number of linear regions in deep neural networks, providing a practical measure of their expressiveness and explaining the benefits of skip connections and residual structures.

Contribution

It proposes a general computational framework to compute a tighter upper bound of linear regions for various network architectures, improving understanding of network expressiveness.

Findings

The new upper bound is tighter than existing bounds.

Skip connections and residual structures enhance network expressiveness.

The framework applies to diverse network structures, including residual and skip-connected networks.

Abstract

The expressiveness of deep neural network (DNN) is a perspective to understandthe surprising performance of DNN. The number of linear regions, i.e. pieces thata piece-wise-linear function represented by a DNN, is generally used to measurethe expressiveness. And the upper bound of regions number partitioned by a rec-tifier network, instead of the number itself, is a more practical measurement ofexpressiveness of a rectifier DNN. In this work, we propose a new and tighter up-per bound of regions number. Inspired by the proof of this upper bound and theframework of matrix computation in Hinz & Van de Geer (2019), we propose ageneral computational approach to compute a tight upper bound of regions numberfor theoretically any network structures (e.g. DNN with all kind of skip connec-tions and residual structures). Our experiments show our upper bound is tighterthan existing ones, and explain why skip connections and residual structures canimprove network performance.