Identifying Suitable Tasks for Inductive Transfer Through the Analysis of Feature Attributions

TL;DR

This paper proposes a method using explainability techniques to predict beneficial task pairs for transfer learning, significantly reducing the need for trial-and-error and computational resources.

Contribution

It introduces a novel approach to predict task complementarity in transfer learning by analyzing neural network activation patterns, avoiding extensive grid searches.

Findings

Reduced training time by up to 83.5%.

Achieved only a 0.034 decrease in positive-class F1 score.

Effective prediction of task transferability without full experiments.

Abstract

Transfer learning approaches have shown to significantly improve performance on downstream tasks. However, it is common for prior works to only report where transfer learning was beneficial, ignoring the significant trial-and-error required to find effective settings for transfer. Indeed, not all task combinations lead to performance benefits, and brute-force searching rapidly becomes computationally infeasible. Hence the question arises, can we predict whether transfer between two tasks will be beneficial without actually performing the experiment? In this paper, we leverage explainability techniques to effectively predict whether task pairs will be complementary, through comparison of neural network activation between single-task models. In this way, we can avoid grid-searches over all task and hyperparameter combinations, dramatically reducing the time needed to find effective task pairs. Our results show that, through this approach, it is possible to reduce training time by up to 83.5% at a cost of only 0.034 reduction in positive-class F1 on the TREC-IS 2020-A dataset.