Network-size independent covering number bounds for deep networks

TL;DR

This paper presents a novel covering number bound for deep neural networks that does not depend on network size, achieved by focusing on data scaling rather than network dimensions.

Contribution

The authors introduce a size-independent covering number bound for deep networks by leveraging data rotation invariance in linear classifiers.

Findings

Covering number bound is independent of network size.

Rotation invariance simplifies the analysis of deep networks.

The approach applies to linear classifiers within deep networks.

Abstract

We give a covering number bound for deep learning networks that is independent of the size of the network. The key for the simple analysis is that for linear classifiers, rotating the data doesn't affect the covering number. Thus, we can ignore the rotation part of each layer's linear transformation, and get the covering number bound by concentrating on the scaling part.