Multi-Task Zipping via Layer-wise Neuron Sharing
Xiaoxi He, Zimu Zhou, Lothar Thiele

TL;DR
This paper introduces Multi-Task Zipping (MTZ), a novel framework for cross-model neural network compression that merges correlated pre-trained models through layer-wise neuron sharing, significantly reducing redundancy and retraining effort.
Contribution
MTZ is the first method to automatically merge multiple pre-trained neural networks across models, enabling efficient cross-model compression with minimal accuracy loss and retraining.
Findings
Successfully merged two VGG-16 networks with minimal accuracy loss.
Shared 39.61% parameters between models, reducing storage.
Retraining time is at least 17.8 times shorter than training a new network.
Abstract
Future mobile devices are anticipated to perceive, understand and react to the world on their own by running multiple correlated deep neural networks on-device. Yet the complexity of these neural networks needs to be trimmed down both within-model and cross-model to fit in mobile storage and memory. Previous studies focus on squeezing the redundancy within a single neural network. In this work, we aim to reduce the redundancy across multiple models. We propose Multi-Task Zipping (MTZ), a framework to automatically merge correlated, pre-trained deep neural networks for cross-model compression. Central in MTZ is a layer-wise neuron sharing and incoming weight updating scheme that induces a minimal change in the error function. MTZ inherits information from each model and demands light retraining to re-boost the accuracy of individual tasks. Evaluations show that MTZ is able to fully merge…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Advanced Image and Video Retrieval Techniques
