Gap Minimization for Knowledge Sharing and Transfer

TL;DR

This paper introduces the performance gap as a new measure for understanding task similarities in transfer and multitask learning, and proposes algorithms that minimize this gap to improve learning outcomes.

Contribution

It defines the performance gap as a novel, data- and algorithm-dependent measure, and develops two algorithms, gapBoost and gapMTNN, based on gap minimization for transfer and multitask learning.

Findings

gapBoost outperforms existing transfer learning methods

gapMTNN achieves superior results in multitask learning benchmarks

The performance gap provides a new theoretical insight into task relatedness

Abstract

Learning from multiple related tasks by knowledge sharing and transfer has become increasingly relevant over the last two decades. In order to successfully transfer information from one task to another, it is critical to understand the similarities and differences between the domains. In this paper, we introduce the notion of \emph{performance gap}, an intuitive and novel measure of the distance between learning tasks. Unlike existing measures which are used as tools to bound the difference of expected risks between tasks (e.g., $\mathcal{H}$-divergence or discrepancy distance), we theoretically show that the performance gap can be viewed as a data- and algorithm-dependent regularizer, which controls the model complexity and leads to finer guarantees. More importantly, it also provides new insights and motivates a novel principle for designing strategies for knowledge sharing and transfer: gap minimization. We instantiate this principle with two algorithms: 1. gapBoost, a novel and principled boosting algorithm that explicitly minimizes the performance gap between source and target domains for transfer learning; and 2. gapMTNN, a representation learning algorithm that reformulates gap minimization as semantic conditional matching for multitask learning. Our extensive evaluation on both transfer learning and multitask learning benchmark data sets shows that our methods outperform existing baselines.