Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks

TL;DR

This paper introduces tangent sensitivity to measure the stability of ReLU neural networks concerning small parameter changes and relates it to the distribution of activation regions, providing bounds and empirical insights into generalization.

Contribution

It proposes tangent sensitivity as a new measure for neural network stability and connects it to activation region distribution, with practical bounds and empirical analysis.

Findings

Tangent sensitivity correlates with the generalization gap.

Simple bounds effectively capture network stability.

Distribution of activation regions influences model robustness.

Abstract

Recent articles indicate that deep neural networks are efficient models for various learning problems. However they are often highly sensitive to various changes that cannot be detected by an independent observer. As our understanding of deep neural networks with traditional generalization bounds still remains incomplete, there are several measures which capture the behaviour of the model in case of small changes at a specific state. In this paper we consider adversarial stability in the tangent space and suggest tangent sensitivity in order to characterize stability. We focus on a particular kind of stability with respect to changes in parameters that are induced by individual examples without known labels. We derive several easily computable bounds and empirical measures for feed-forward fully connected ReLU (Rectified Linear Unit) networks and connect tangent sensitivity to the distribution of the activation regions in the input space realized by the network. Our experiments suggest that even simple bounds and measures are associated with the empirical generalization gap.