Using accumulation to optimize deep residual neural nets

TL;DR

This paper introduces a novel residual connection method in deep neural networks that incorporates residuals from all lower layers, leading to improved performance on CIFAR-10.

Contribution

It proposes a new accumulation-based residual approach that enhances deep residual neural networks by integrating all previous layers' residuals.

Findings

Improved accuracy on CIFAR-10 dataset

Enhanced contribution of all layers to the output

Potential for better deep network training

Abstract

Residual Neural Networks [1] won first place in all five main tracks of the ImageNet and COCO 2015 competitions. This kind of network involves the creation of pluggable modules such that the output contains a residual from the input. The residual in that paper is the identity function. We propose to include residuals from all lower layers, suitably normalized, to create the residual. This way, all previous layers contribute equally to the output of a layer. We show that our approach is an improvement on [1] for the CIFAR-10 dataset.