Net2Net: Accelerating Learning via Knowledge Transfer

TL;DR

Net2Net introduces function-preserving transformations that enable rapid transfer of knowledge between neural networks, significantly accelerating training and improving accuracy on large-scale datasets.

Contribution

The paper presents a novel method for instantaneously transferring learned representations to larger neural networks, reducing training time during experimentation.

Findings

Enables rapid transfer of knowledge between neural networks.

Achieves state-of-the-art accuracy on ImageNet with transferred models.

Reduces training time during neural network design and experimentation.

Abstract

We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.