Knowledge Flow: Improve Upon Your Teachers

TL;DR

This paper introduces a knowledge flow method that transfers knowledge from multiple deep teacher networks to a student network, enabling effective learning across different tasks and architectures, outperforming existing fine-tuning methods.

Contribution

It proposes a novel knowledge flow technique allowing knowledge transfer between diverse models and tasks, independent of their structure or output spaces.

Findings

Student models outperform fine-tuning methods.

Knowledge flow works across supervised and reinforcement learning.

Models achieve better generalization after knowledge transfer.

Abstract

A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.