Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning

TL;DR

This paper introduces Embedding Space Maps (ESMs), a lightweight neural network approach that efficiently predicts the transferability of NLP tasks, enabling large-scale task selection with reduced computational resources.

Contribution

The authors propose ESMs to approximate fine-tuning effects, allowing scalable and resource-efficient task transferability ranking in NLP.

Findings

ESMs reduce execution time by a factor of 10.

ESMs decrease disk space usage by a factor of 278.

High transferability prediction accuracy with an average regret@5 score of 2.95.

Abstract

Intermediate task transfer learning can greatly improve model performance. If, for example, one has little training data for emotion detection, first fine-tuning a language model on a sentiment classification dataset may improve performance strongly. But which task to choose for transfer learning? Prior methods producing useful task rankings are infeasible for large source pools, as they require forward passes through all source language models. We overcome this by introducing Embedding Space Maps (ESMs), light-weight neural networks that approximate the effect of fine-tuning a language model. We conduct the largest study on NLP task transferability and task selection with 12k source-target pairs. We find that applying ESMs on a prior method reduces execution time and disk space usage by factors of 10 and 278, respectively, while retaining high selection performance (avg. regret@5 score of 2.95).