Efficiently Tuned Parameters are Task Embeddings

TL;DR

This paper introduces a simple, efficient method for predicting task transferability in NLP by using task-specific parameters from parameter-efficient tuning as task embeddings, improving source dataset selection.

Contribution

It proposes leveraging efficiently tuned parameters as task embeddings for better transferability prediction, enabling more effective and computationally efficient source dataset selection.

Findings

Outperforms existing transferability prediction methods.

Task embeddings from early checkpoints improve efficiency.

Transferability prediction is disentangled from in-task performance.

Abstract

Intermediate-task transfer can benefit a wide range of NLP tasks with properly selected source datasets. However, it is computationally infeasible to experiment with all intermediate transfer combinations, making choosing a useful source task a challenging problem. In this paper, we anticipate that task-specific parameters updated in parameter-efficient tuning methods are likely to encode task-specific information. Therefore, such parameters can be predictive for inter-task transferability. Thus, we propose to exploit these efficiently tuned parameters as off-the-shelf task embeddings for the efficient selection of source datasets for intermediate-task transfer. We experiment with 11 text classification tasks and 11 question answering tasks. Experimental results show that our approach can consistently outperform existing inter-task transferability prediction methods while being conceptually simple and computationally efficient. Our analysis also reveals that the ability of efficiently tuned parameters on transferability prediction is disentangled with their in-task performance. This allows us to use parameters from early checkpoints as task embeddings to further improve efficiency.