Go Wide, Then Narrow: Efficient Training of Deep Thin Networks

TL;DR

This paper introduces an efficient three-stage training method for deep thin networks, leveraging wide networks for initialization and layerwise imitation, backed by theoretical guarantees and empirical validation.

Contribution

It proposes a novel training approach that improves deep thin network performance by using wide networks for initialization and layerwise imitation, with theoretical analysis and large-scale experiments.

Findings

ResNet50 with our method outperforms ResNet101.

BERT Base becomes comparable to BERT Large.

Theoretical guarantee via neural mean field analysis.

Abstract

For deploying a deep learning model into production, it needs to be both accurate and compact to meet the latency and memory constraints. This usually results in a network that is deep (to ensure performance) and yet thin (to improve computational efficiency). In this paper, we propose an efficient method to train a deep thin network with a theoretic guarantee. Our method is motivated by model compression. It consists of three stages. First, we sufficiently widen the deep thin network and train it until convergence. Then, we use this well-trained deep wide network to warm up (or initialize) the original deep thin network. This is achieved by layerwise imitation, that is, forcing the thin network to mimic the intermediate outputs of the wide network from layer to layer. Finally, we further fine tune this already well-initialized deep thin network. The theoretical guarantee is established by using the neural mean field analysis. It demonstrates the advantage of our layerwise imitation approach over backpropagation. We also conduct large-scale empirical experiments to validate the proposed method. By training with our method, ResNet50 can outperform ResNet101, and BERT Base can be comparable with BERT Large, when ResNet101 and BERT Large are trained under the standard training procedures as in the literature.