Residual Networks Behave Like Boosting Algorithms

TL;DR

This paper reveals that Residual Networks function similarly to boosting algorithms, providing theoretical bounds and proposing novel boosting-inspired neural network and decision tree methods with competitive performance.

Contribution

It establishes a theoretical equivalence between ResNet and boosting, introduces architectural modifications for boosting, and develops new boosting-inspired algorithms with proven error bounds.

Findings

ResNet is equivalent to boosting feature representation.

Proposed residual modules with max-norm bounds improve boosting.

Online boosted decision trees perform comparably to state-of-the-art offline methods.

Abstract

We show that Residual Networks (ResNet) is equivalent to boosting feature representation, without any modification to the underlying ResNet training algorithm. A regret bound based on Online Gradient Boosting theory is proved and suggests that ResNet could achieve Online Gradient Boosting regret bounds through neural network architectural changes with the addition of a shrinkage parameter in the identity skip-connections and using residual modules with max-norm bounds. Through this relation between ResNet and Online Boosting, novel feature representation boosting algorithms can be constructed based on altering residual modules. We demonstrate this through proposing decision tree residual modules to construct a new boosted decision tree algorithm and demonstrating generalization error bounds for both approaches; relaxing constraints within BoostResNet algorithm to allow it to be trained in an out-of-core manner. We evaluate convolution ResNet with and without shrinkage modifications to demonstrate its efficacy, and demonstrate that our online boosted decision tree algorithm is comparable to state-of-the-art offline boosted decision tree algorithms without the drawback of offline approaches.