Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements

TL;DR

This paper introduces a Generalization Stress Test revealing that large language models, despite high benchmark scores, struggle with minor content-preserving modifications, exposing reliance on superficial cues rather than robust understanding.

Contribution

The study presents a novel stress test framework to evaluate LLMs' generalization abilities under controlled perturbations, highlighting their limitations and biases.

Findings

LLMs show significant accuracy drops with minor content changes.

High benchmark scores do not guarantee robust generalization.

Models rely heavily on superficial cues rather than deep understanding.

Abstract

In this paper, we propose a ``Generalization Stress Test" to assess Large Language Models' (LLMs) generalization ability under slight and controlled perturbations, including option length, problem types, and irrelevant noun replacements. We achieve novel and significant findings that, despite high benchmark scores, LLMs exhibit severe accuracy drops and unexpected biases (e.g., preference for longer distractors) when faced with these minor but content-preserving modifications. For example, Qwen 2.5 1.5B's MMLU score rises from 60 to 89 and drops from 89 to 36 when option lengths are changed without altering the question. Even GPT4o experiences a 25-point accuracy loss when problem types are changed, with a 6-point drop across all three modification categories. These analyses suggest that LLMs rely heavily on superficial cues rather than forming robust, abstract representations that generalize across formats, lexical variations, and irrelevant content shifts.