Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?

TL;DR

This study investigates when and why intermediate-task training improves pretrained language models' performance on natural language understanding tasks, revealing that tasks requiring high-level reasoning are most beneficial.

Contribution

It provides a large-scale analysis of intermediate-task transfer with RoBERTa, identifying key skills that enhance transfer learning effectiveness.

Findings

High-level inference tasks improve transfer performance.

Target task success correlates with coreference resolution abilities.

Forgetting pretraining knowledge may limit transfer improvements.

Abstract

While pretrained models such as BERT have shown large gains across natural language understanding tasks, their performance can be improved by further training the model on a data-rich intermediate task, before fine-tuning it on a target task. However, it is still poorly understood when and why intermediate-task training is beneficial for a given target task. To investigate this, we perform a large-scale study on the pretrained RoBERTa model with 110 intermediate-target task combinations. We further evaluate all trained models with 25 probing tasks meant to reveal the specific skills that drive transfer. We observe that intermediate tasks requiring high-level inference and reasoning abilities tend to work best. We also observe that target task performance is strongly correlated with higher-level abilities such as coreference resolution. However, we fail to observe more granular correlations between probing and target task performance, highlighting the need for further work on broad-coverage probing benchmarks. We also observe evidence that the forgetting of knowledge learned during pretraining may limit our analysis, highlighting the need for further work on transfer learning methods in these settings.