How to Train Your Metamorphic Deep Neural Network

TL;DR

This paper introduces an improved training algorithm for Neural Metamorphosis (NeuMeta), enabling full-network metamorphosis with minimal accuracy loss, thus supporting scalable and efficient deployment of adaptable deep neural networks.

Contribution

It extends NeuMeta to enable full-network metamorphosis through a structured training approach, broadening its applicability beyond final layers.

Findings

Maintains competitive accuracy across various compression ratios.

Enables full-network metamorphosis with minimal accuracy degradation.

Provides a scalable solution for adaptable neural network deployment.

Abstract

Neural Metamorphosis (NeuMeta) is a recent paradigm for generating neural networks of varying width and depth. Based on Implicit Neural Representation (INR), NeuMeta learns a continuous weight manifold, enabling the direct generation of compressed models, including those with configurations not seen during training. While promising, the original formulation of NeuMeta proves effective only for the final layers of the undelying model, limiting its broader applicability. In this work, we propose a training algorithm that extends the capabilities of NeuMeta to enable full-network metamorphosis with minimal accuracy degradation. Our approach follows a structured recipe comprising block-wise incremental training, INR initialization, and strategies for replacing batch normalization. The resulting metamorphic networks maintain competitive accuracy across a wide range of compression ratios, offering a scalable solution for adaptable and efficient deployment of deep models. The code is available at: https://github.com/TSommariva/HTTY_NeuMeta.