An information-Theoretic Approach to Semi-supervised Transfer Learning

TL;DR

This paper introduces an information-theoretic framework for semi-supervised transfer learning, proposing regularization methods based on mutual and Lautum information to enhance transferability in deep neural networks.

Contribution

It develops novel information-theoretic regularization techniques for semi-supervised transfer learning, improving neural network transferability under distribution discrepancies.

Findings

Regularization based on mutual information improves transfer performance.

Lautum information-based regularization enhances learning with unlabeled target data.

Proposed methods outperform baseline transfer learning approaches.

Abstract

Transfer learning is a valuable tool in deep learning as it allows propagating information from one "source dataset" to another "target dataset", especially in the case of a small number of training examples in the latter. Yet, discrepancies between the underlying distributions of the source and target data are commonplace and are known to have a substantial impact on algorithm performance. In this work we suggest novel information-theoretic approaches for the analysis of the performance of deep neural networks in the context of transfer learning. We focus on the task of semi-supervised transfer learning, in which unlabeled samples from the target dataset are available during network training on the source dataset. Our theory suggests that one may improve the transferability of a deep neural network by incorporating regularization terms on the target data based on information-theoretic quantities, namely the Mutual Information and the Lautum Information. We demonstrate the effectiveness of the proposed approaches in various semi-supervised transfer learning experiments.