Parameter-Efficient Tuning Makes a Good Classification Head

TL;DR

This paper shows that parameter-efficient tuning methods can create effective classification heads for pretrained models, leading to stable performance improvements across multiple NLP tasks without full finetuning.

Contribution

It introduces the idea that parameter-efficient tuning can produce good classification heads, reducing the need for full model finetuning and enhancing stability.

Findings

Pretrained classification heads via parameter-efficient tuning improve performance.

The approach is effective across 9 GLUE and SuperGLUE tasks.

Stable performance gains are achieved without full finetuning.

Abstract

In recent years, pretrained models revolutionized the paradigm of natural language understanding (NLU), where we append a randomly initialized classification head after the pretrained backbone, e.g. BERT, and finetune the whole model. As the pretrained backbone makes a major contribution to the improvement, we naturally expect a good pretrained classification head can also benefit the training. However, the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning, making the usual head-only pretraining (LP-FT) ineffective. In this paper, we find that parameter-efficient tuning makes a good classification head, with which we can simply replace the randomly initialized heads for a stable performance gain. Our experiments demonstrate that the classification head jointly pretrained with parameter-efficient tuning consistently improves the performance on 9 tasks in GLUE and SuperGLUE.