Why ResNet Works? Residuals Generalize

TL;DR

This paper provides a theoretical analysis showing that residual connections do not increase hypothesis complexity and offers a generalization bound for ResNet, supporting the effectiveness of residuals in deep neural networks.

Contribution

It proves residuals do not increase hypothesis complexity and derives a generalization bound for ResNet, linking residuals to improved generalization.

Findings

Residual connections do not increase hypothesis complexity compared to chain-like networks.

A $ ext{O}(1 / \sqrt{N})$ generalization bound for ResNet is established.

Regularization of weight norms is crucial for good generalization.

Abstract

Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We prove that the upper bound of the covering number is the same as chain-like neural networks, if the total numbers of the weight matrices and nonlinearities are fixed, no matter whether they are in the residuals or not. This result demonstrates that residual connections may not increase the hypothesis complexity of the neural network compared with the chain-like counterpart. Based on the upper bound of the covering number, we then obtain an $\mathcal O(1 / \sqrt{N})$ margin-based multi-class generalization bound for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state-of-the-art neural network architectures, such as DenseNet and ResNeXt, are straight-forward. From our generalization bound, a practical implementation is summarized: to approach a good generalization ability, we need to use regularization terms to control the magnitude of the norms of weight matrices not to increase too much, which justifies the standard technique of weight decay.