Breaking the Architecture Barrier: A Method for Efficient Knowledge Transfer Across Networks

TL;DR

This paper introduces DPIAT, a dynamic programming-based method for transferring neural network parameters across different architectures, significantly enhancing training efficiency and accuracy.

Contribution

It presents a novel approach for cross-architecture transfer learning using dynamic programming, enabling effective knowledge reuse without retraining.

Findings

Improved validation accuracy by 1.6x on ImageNet after 50 epochs.

DPIAT outperforms existing parameter prediction and random initialization methods.

Introduces a network architecture similarity measure for source network selection.

Abstract

Transfer learning is a popular technique for improving the performance of neural networks. However, existing methods are limited to transferring parameters between networks with same architectures. We present a method for transferring parameters between neural networks with different architectures. Our method, called DPIAT, uses dynamic programming to match blocks and layers between architectures and transfer parameters efficiently. Compared to existing parameter prediction and random initialization methods, it significantly improves training efficiency and validation accuracy. In experiments on ImageNet, our method improved validation accuracy by an average of 1.6 times after 50 epochs of training. DPIAT allows both researchers and neural architecture search systems to modify trained networks and reuse knowledge, avoiding the need for retraining from scratch. We also introduce a network architecture similarity measure, enabling users to choose the best source network without any training.