Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

TL;DR

This paper investigates the local properties of linear regions in deep neural networks with piecewise linear activations, revealing how different optimization methods influence these regions despite similar accuracy.

Contribution

It introduces a detailed empirical analysis of the local properties of linear regions in DNNs, beyond just counting their number, and explores how optimization affects these regions.

Findings

Different optimization techniques lead to distinct linear regions.

Similar classification accuracy can be achieved with different local region properties.

Insights may inspire new optimization methods and understanding of DNN behaviors.

Abstract

A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.