On decision regions of narrow deep neural networks

TL;DR

This paper proves that narrow deep neural networks with width less than or equal to input dimension have unbounded connected decision regions, extending understanding of their geometric properties and decision boundaries.

Contribution

It establishes that all connected components of decision regions in such narrow networks are unbounded, for various activation functions, complementing existing approximation and connectivity results.

Findings

Connected components of decision regions are unbounded.

Results hold for ReLU and monotonic activation functions.

Numerical experiments illustrate theoretical findings.

Abstract

We show that for neural network functions that have width less or equal to the input dimension all connected components of decision regions are unbounded. The result holds for continuous and strictly monotonic activation functions as well as for the ReLU activation function. This complements recent results on approximation capabilities by [Hanin 2017 Approximating] and connectivity of decision regions by [Nguyen 2018 Neural] for such narrow neural networks. Our results are illustrated by means of numerical experiments.