Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement

TL;DR

This paper introduces TRED, a novel transfer learning method that disentangles target-relevant knowledge from source models to improve fine-tuning performance, effectively reducing negative transfer risks.

Contribution

The paper proposes TRED, a new transfer learning algorithm that uses representation disentanglement with Max-MMD and Min-MI to enhance fine-tuning accuracy.

Findings

TRED improves fine-tuning accuracy by over 2% on average.

TRED outperforms state-of-the-art regularizers like L2-SP, AT, DELTA, and BSS.

Representation disentanglement effectively reduces negative transfer.

Abstract

Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples. To obtain better generalization, using the starting point as the reference (SPAR), either through weights or features, has been successfully applied to transfer learning as a regularizer. However, due to the domain discrepancy between the source and target task, there exists obvious risk of negative transfer in a straightforward manner of knowledge preserving. In this paper, we propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED), where the relevant knowledge with respect to the target task is disentangled from the original source model and used as a regularizer during fine-tuning the target model. Specifically, we design two alternative methods, maximizing the Maximum Mean Discrepancy (Max-MMD) and minimizing the mutual information (Min-MI), for the representation disentanglement. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average. TRED also outperforms related state-of-the-art transfer learning regularizers such as L2-SP, AT, DELTA, and BSS.