Identity Mappings in Deep Residual Networks

TL;DR

This paper analyzes the role of identity mappings in deep residual networks, demonstrating their importance for signal propagation and proposing a new residual unit that enhances training and generalization, leading to improved results on multiple datasets.

Contribution

The paper introduces a new residual unit leveraging identity mappings, improving training ease and generalization in very deep ResNets.

Findings

Improved accuracy with 1001-layer ResNet on CIFAR-10 and CIFAR-100

Enhanced training stability and convergence

Successful deployment of 200-layer ResNet on ImageNet

Abstract

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers