Learning Deep ResNet Blocks Sequentially using Boosting Theory

TL;DR

This paper introduces BoostResNet, a boosting-based training method for deep ResNets that constructs the network from sequentially trained shallow modules, with theoretical guarantees on error decay and generalization.

Contribution

It develops a boosting theory for ResNet architectures, enabling a new training algorithm suitable for non-differentiable models and providing theoretical error and generalization bounds.

Findings

Training error decays exponentially with depth T.

BoostResNet performs well under weak learning conditions.

ResNet's generalization is resistant to overfitting with bounded weights.

Abstract

Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep \emph{residual network} (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct $T$ weak module classifiers, each contains two of the $T$ layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, \emph{BoostResNet}, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of $T$ "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth $T$ if the \emph{weak module classifiers} that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with $l_1$ norm bounded weights.