Does QA-based intermediate training help fine-tuning language models for text classification?

TL;DR

This study investigates whether intermediate training on QA tasks improves fine-tuning of language models for text classification, revealing variable benefits across models and tasks.

Contribution

It provides empirical evidence on the effects of QA-based intermediate training across multiple models and classification tasks, highlighting inconsistent transfer performance.

Findings

QA training benefits vary across models

Similar QA tasks lead to better transfer performance

Transfer performance is inconsistent across different models

Abstract

Fine-tuning pre-trained language models for downstream tasks has become a norm for NLP. Recently it is found that intermediate training based on high-level inference tasks such as Question Answering (QA) can improve the performance of some language models for target tasks. However it is not clear if intermediate training generally benefits various language models. In this paper, using the SQuAD-2.0 QA task for intermediate training for target text classification tasks, we experimented on eight tasks for single-sequence classification and eight tasks for sequence-pair classification using two base and two compact language models. Our experiments show that QA-based intermediate training generates varying transfer performance across different language models, except for similar QA tasks.