Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks

TL;DR

This paper compares various training procedures for adapting large language models to text classification, highlighting trade-offs between performance and efficiency, and proposing combined prompting and knowledge distillation as a cost-effective approach.

Contribution

It systematically evaluates fine-tuning, prompting, and knowledge distillation, revealing that prompting with KD offers a more efficient adaptation method for large language models.

Findings

Fine-tuning and prompting perform well on large datasets.

Prompting with knowledge distillation reduces compute and data costs.

Alternative training methods can be more efficient without sacrificing accuracy.

Abstract

Pre-trained language models (LMs) obtain state-of-the-art performance when adapted to text classification tasks. However, when using such models in real-world applications, efficiency considerations are paramount. In this paper, we study how different training procedures that adapt LMs to text classification perform, as we vary model and train set size. More specifically, we compare standard fine-tuning, prompting, and knowledge distillation (KD) when the teacher was trained with either fine-tuning or prompting. Our findings suggest that even though fine-tuning and prompting work well to train large LMs on large train sets, there are more efficient alternatives that can reduce compute or data cost. Interestingly, we find that prompting combined with KD can reduce compute and data cost at the same time.