FractalNet: Ultra-Deep Neural Networks without Residuals

TL;DR

This paper presents FractalNet, a novel deep neural network architecture based on self-similar fractal structures that achieves competitive performance without residual connections, highlighting the importance of transitioning from shallow to deep during training.

Contribution

Introduces fractal-based macro-architecture for neural networks, demonstrating competitive results without residuals and proposing drop-path regularization for subpath co-adaptation.

Findings

FractalNet matches residual networks on CIFAR and ImageNet.

Regularization via drop-path improves subpath training.

Fractal networks have an anytime property with shallow and deep subnetworks.

Abstract

We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.