Making Pretrained Language Models Good Long-tailed Learners

TL;DR

This paper investigates prompt-tuning's effectiveness for long-tailed classification, demonstrating it enables pretrained language models to perform well on imbalanced data, with analysis highlighting the importance of classifier structure over input design.

Contribution

It provides empirical evidence that prompt-tuning makes pretrained models effective long-tailed learners and analyzes the underlying reasons, emphasizing classifier structure importance.

Findings

Prompt-tuning improves long-tailed classification performance.

Classifier structure and parameterization are key to success.

Findings extend to few-shot classification scenarios.

Abstract

Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners. For intuitions on why prompt-tuning can achieve good performance in long-tailed classification, we carry out in-depth analyses by progressively bridging the gap between prompt-tuning and commonly used finetuning. The summary is that the classifier structure and parameterization form the key to making good long-tailed learners, in comparison with the less important input structure. Finally, we verify the applicability of our finding to few-shot classification. Good long-tailed learners can be abbreviated as Glee.