Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning

TL;DR

This paper investigates how the choice of intermediate tasks affects transfer learning performance, revealing high variance across tasks and proposing new methods for better task selection using token similarity.

Contribution

It introduces a novel token similarity-based method for task prediction and compares various task selection strategies, highlighting the importance of task embeddings and transferability estimation.

Findings

Transfer performance varies significantly across source tasks and seeds.

Task embeddings improve transferability prediction from 2.59% to 3.96%.

Token-wise similarity outperforms weight averaging in predicting transferability.

Abstract

Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. In this work, we experiment with 130 source-target task combinations and demonstrate that the transfer performance exhibits severe variance across different source tasks and training seeds, highlighting the crucial role of intermediate-task selection in a broader context. We compare four representative task selection methods in a unified setup, focusing on their effectiveness and consistency. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability by improving task prediction scores from 2.59% to 3.96%. Despite their strong performance, we observe that the task embeddings do not consistently demonstrate superiority for tasks requiring reasoning abilities. Furthermore, we introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction. Our findings suggest that token-wise similarity is better predictive for predicting transferability compared to averaging weights.