Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts

TL;DR

This paper investigates how large language models perform on negated prompts, revealing an inverse scaling law where larger models perform worse, highlighting a critical limitation and urging development of better instruction-following methods.

Contribution

It demonstrates that large LMs do not follow the expected scaling law on negated prompts and introduces a comprehensive evaluation of various models on this challenge.

Findings

Larger LMs perform worse on negated prompts

Significant performance gap between models and humans on negated prompts

Existing models struggle to follow negation instructions accurately

Abstract

Previous work has shown that there exists a scaling law between the size of Language Models (LMs) and their zero-shot performance on different downstream NLP tasks. In this work, we show that this phenomenon does not hold when evaluating large LMs on tasks with negated prompts, but instead shows an inverse scaling law. We evaluate 9 different tasks with negated prompts on (1) pretrained LMs (OPT & GPT-3) of varying sizes (125M - 175B), (2) LMs further pretrained to generalize to novel prompts (InstructGPT), (3) LMs provided with few-shot examples, and (4) LMs fine-tuned specifically on negated prompts; all LM types perform worse on negated prompts as they scale and show a huge performance gap between the human performance when comparing the average score on both original and negated prompts. By highlighting a critical limitation of existing LMs and methods, we urge the community to develop new approaches of developing LMs that actually follow the given instructions. We provide the code and the datasets to explore negated prompts at https://github.com/joeljang/negated-prompts-for-llms