Network Recasting: A Universal Method for Network Architecture Transformation
Joonsang Yu, Sungbum Kang, Kiyoung Choi

TL;DR
This paper introduces network recasting, a versatile method for transforming neural network architectures to accelerate inference and enable compression, by sequentially approximating each block of a teacher network with a target network.
Contribution
It presents a novel block-wise recasting technique that can transform any network architecture into another, including mixed architectures, while maintaining accuracy and improving inference speed.
Findings
Outperforms previous methods in actual GPU speedup
Can generate mixed-architecture networks
Effectively reduces parameters and activations
Abstract
This paper proposes network recasting as a general method for network architecture transformation. The primary goal of this method is to accelerate the inference process through the transformation, but there can be many other practical applications. The method is based on block-wise recasting; it recasts each source block in a pre-trained teacher network to a target block in a student network. For the recasting, a target block is trained such that its output activation approximates that of the source block. Such a block-by-block recasting in a sequential manner transforms the network architecture while preserving the accuracy. This method can be used to transform an arbitrary teacher network type to an arbitrary student network type. It can even generate a mixed-architecture network that consists of two or more types of block. The network recasting can generate a network with fewer…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Memory and Neural Computing · Neural Networks and Applications · Advanced Neural Network Applications
