Understanding How Model Size Affects Few-shot Instruction Prompting

TL;DR

This paper investigates how model size influences the ability of large language models to perform few-shot instruction tasks, revealing a weak inverse scaling trend and the impact of example quantity on performance.

Contribution

It introduces DeltaWords, a new dataset for evaluating model understanding of word antonyms, and analyzes how model size and example count affect task accuracy.

Findings

Accuracy decreases slightly as model size increases in few-shot settings.

Adding more examples benefits larger models more than smaller ones.

Weak inverse scaling trend observed across model sizes.

Abstract

Large Language Models are affected by the phenomena of memorizing and forgetting their training data. But how do these vary by model size? We work towards this question by investigating how the model size affects the model's ability to discriminate a word's meaning in a given context. We introduce a dataset called DeltaWords, which evaluates a model's ability to follow instructions to select a sentence which replaces the target word with its antonym. We show a weak inverse scaling trend, where task accuracy degrades as model size increase, under extremely few-shot prompting regimes. We show that increasing the number of examples tend to disproportionately benefit larger models than smaller models.