What to Pre-Train on? Efficient Intermediate Task Selection

TL;DR

This paper demonstrates that embedding-based methods can efficiently identify beneficial intermediate tasks for transfer learning in NLP, achieving near-optimal results with minimal computational cost.

Contribution

It consolidates and evaluates efficient dataset selection methods for intermediate transfer learning, outperforming expensive few-shot approaches.

Findings

Embedding-based methods outperform few-shot fine-tuning approaches.

Achieve less than 1% average Regret@3 in task selection.

Effective across diverse NLP tasks.

Abstract

Intermediate task fine-tuning has been shown to culminate in large transfer gains across many NLP tasks. With an abundance of candidate datasets as well as pre-trained language models, it has become infeasible to run the cross-product of all combinations to find the best transfer setting. In this work we first establish that similar sequential fine-tuning gains can be achieved in adapter settings, and subsequently consolidate previously proposed methods that efficiently identify beneficial tasks for intermediate transfer learning. We experiment with a diverse set of 42 intermediate and 11 target English classification, multiple choice, question answering, and sequence tagging tasks. Our results show that efficient embedding based methods that rely solely on the respective datasets outperform computational expensive few-shot fine-tuning approaches. Our best methods achieve an average Regret@3 of less than 1% across all target tasks, demonstrating that we are able to efficiently identify the best datasets for intermediate training.