Radius-margin bounds for deep neural networks

TL;DR

This paper extends radius-margin bounds from SVMs to deep neural networks, providing new capacity measures and insights into techniques like Dropout and margin maximization for robustness.

Contribution

It introduces radius-margin bounds for deep networks, relating capacity to network features and analyzing the impact of regularization techniques.

Findings

Radius-margin bounds are applicable to deep architectures.

Dropout and Dropconnect reduce network capacity.

Maximizing margins enhances robustness against input noise.

Abstract

Explaining the unreasonable effectiveness of deep learning has eluded researchers around the globe. Various authors have described multiple metrics to evaluate the capacity of deep architectures. In this paper, we allude to the radius margin bounds described for a support vector machine (SVM) with hinge loss, apply the same to the deep feed-forward architectures and derive the Vapnik-Chervonenkis (VC) bounds which are different from the earlier bounds proposed in terms of number of weights of the network. In doing so, we also relate the effectiveness of techniques like Dropout and Dropconnect in bringing down the capacity of the network. Finally, we describe the effect of maximizing the input as well as the output margin to achieve an input noise-robust deep architecture.