Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning

TL;DR

This paper introduces a method for knowledge amalgamation from heterogeneous pre-trained networks by transforming their features into a common space, enabling a lightweight student model to learn integrated knowledge without human annotations.

Contribution

It proposes a novel common feature learning scheme for transferring knowledge from diverse architectures into a single student model without annotations.

Findings

Student model outperforms individual teachers on benchmarks.

The approach effectively integrates heterogeneous network knowledge.

Achieves superior performance without access to training data.

Abstract

An increasing number of well-trained deep networks have been released online by researchers and developers, enabling the community to reuse them in a plug-and-play way without accessing the training annotations. However, due to the large number of network variants, such public-available trained models are often of different architectures, each of which being tailored for a specific task or dataset. In this paper, we study a deep-model reusing task, where we are given as input pre-trained networks of heterogeneous architectures specializing in distinct tasks, as teacher models. We aim to learn a multitalented and light-weight student model that is able to grasp the integrated knowledge from all such heterogeneous-structure teachers, again without accessing any human annotation. To this end, we propose a common feature learning scheme, in which the features of all teachers are transformed into a common space and the student is enforced to imitate them all so as to amalgamate the intact knowledge. We test the proposed approach on a list of benchmarks and demonstrate that the learned student is able to achieve very promising performance, superior to those of the teachers in their specialized tasks.