Towards a More Complete Theory of Function Preserving Transforms

TL;DR

This paper introduces R2R, a novel function preserving transform that incorporates residual connections, enabling more flexible neural network architecture modifications and faster training, with improved diversity in learned filters.

Contribution

The paper develops R2R, a new method for function preserving transforms that includes residual connections, expanding the applicability and effectiveness of architecture transformations.

Findings

R2R achieves competitive performance with existing transforms.

R2R enables faster training of neural networks.

R2R learns more diverse filters in image classification tasks.

Abstract

In this paper, we develop novel techniques that can be used to alter the architecture of a neural network, while maintaining the function it represents. Such operations are known as function preserving transforms and have proven useful in transferring knowledge between networks to evaluate architectures quickly, thus having applications in efficient architectures searches. Our methods allow the integration of residual connections into function preserving transforms, so we call them R2R. We provide a derivation for R2R and show that it yields competitive performance with other function preserving transforms, thereby decreasing the restrictions on deep learning architectures that can be extended through function preserving transforms. We perform a comparative analysis with other function preserving transforms such as Net2Net and Network Morphisms, where we shed light on their differences and individual use cases. Finally, we show the effectiveness of R2R to train models quickly, as well as its ability to learn a more diverse set of filters on image classification tasks compared to Net2Net and Network Morphisms.