DiracNets: Training Very Deep Neural Networks Without Skip-Connections

TL;DR

DiracNets introduce a simple parameterization that enables training very deep plain neural networks without skip-connections, matching the performance of ResNets while reducing complexity and initialization sensitivity.

Contribution

The paper proposes Dirac weight parameterization, allowing deep plain networks to perform as well as ResNets without explicit skip-connections.

Findings

DiracNets match ResNet-1001 accuracy on CIFAR-10 with 28-layer plain networks.

DiracNets closely match ResNet performance on ImageNet.

The parameterization reduces the need for careful initialization.

Abstract

Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them - training deeper networks - does not actually hold true, and the benefits come from increased capacity, rather than from depth. Motivated by this, and inspired from ResNet, we propose a simple Dirac weight parameterization, which allows us to train very deep plain networks without explicit skip-connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at inference, as both Dirac parameterization and batch normalization can be folded into convolutional filters, so that network becomes a simple chain of convolution-ReLU pairs. We are able to match ResNet-1001 accuracy on CIFAR-10 with 28-layer wider plain DiracNet, and closely match ResNets on ImageNet. Our parameterization also mostly eliminates the need of careful initialization in residual and non-residual networks. The code and models for our experiments are available at https://github.com/szagoruyko/diracnets