Stable ResNet
Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis,, Arnaud Doucet, Judith Rousseau
TL;DR
This paper introduces Stable ResNet, a new architecture designed to maintain stable gradients and expressivity even as the network depth approaches infinity, addressing key limitations of traditional ResNets.
Contribution
The paper proposes Stable ResNet architectures that stabilize gradients and preserve expressivity in very deep networks, improving upon existing ResNet models.
Findings
Stable ResNet stabilizes gradients in deep networks.
Stable ResNet maintains expressivity at infinite depth.
The architecture outperforms traditional ResNets in stability and expressivity.
Abstract
Deep ResNet architectures have achieved state of the art performance on many tasks. While they solve the problem of gradient vanishing, they might suffer from gradient exploding as the depth becomes large (Yang et al. 2017). Moreover, recent results have shown that ResNet might lose expressivity as the depth goes to infinity (Yang et al. 2017, Hayou et al. 2019). To resolve these issues, we introduce a new class of ResNet architectures, called Stable ResNet, that have the property of stabilizing the gradient while ensuring expressivity in the infinite depth limit.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Human Pose and Action Recognition · Domain Adaptation and Few-Shot Learning
MethodsAverage Pooling · Residual Connection · Convolution · Kaiming Initialization · Global Average Pooling · 1x1 Convolution · Residual Block · *Communicated@Fast*How Do I Communicate to Expedia? · Batch Normalization · Bottleneck Residual Block