Parameter-Efficient Transfer Learning for NLP

TL;DR

This paper introduces adapter modules for transfer learning in NLP, enabling efficient multi-task learning by adding minimal parameters per task while maintaining near state-of-the-art performance.

Contribution

The paper proposes adapter modules as a parameter-efficient alternative to full fine-tuning for NLP transfer learning, demonstrating their effectiveness across multiple tasks.

Findings

Adapters achieve within 0.4% of full fine-tuning performance on GLUE.

Adapters add only 3.6% of parameters per task.

High parameter sharing enables efficient multi-task learning.

Abstract

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.