Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

TL;DR

This paper demonstrates that finetuning language models, especially bias-only finetuning, can replace prompt engineering in few-shot learning, achieving competitive accuracy with minimal parameter updates and improved robustness.

Contribution

It introduces bias-only finetuning as an efficient alternative to prompt engineering for few-shot learning with language models, reducing parameter updates significantly.

Findings

Bias-only finetuning achieves comparable accuracy to full finetuning.

Null prompts with finetuning outperform manually-tuned prompts.

Finetuning only bias terms updates 0.1% of parameters, reducing memory overhead.

Abstract

Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.