How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark

TL;DR

This paper introduces GSM-DC, a synthetic benchmark for evaluating LLM reasoning robustness against irrelevant context, revealing models' sensitivity and proposing training and inference improvements to enhance robustness.

Contribution

The paper presents GSM-DC, a controlled benchmark for systematic evaluation of LLM reasoning under distractors, and proposes training and inference methods to improve robustness.

Findings

LLMs are significantly affected by irrelevant context.

Training with strong distractors improves model robustness.

Stepwise tree search enhances out-of-distribution reasoning robustness.

Abstract

We introduce Grade School Math with Distracting Context (GSM-DC), a synthetic benchmark to evaluate Large Language Models' (LLMs) reasoning robustness against systematically controlled irrelevant context (IC). GSM-DC constructs symbolic reasoning graphs with precise distractor injections, enabling rigorous, reproducible evaluation. Our experiments demonstrate that LLMs are significantly sensitive to IC, affecting both reasoning path selection and arithmetic accuracy. Additionally, training models with strong distractors improves performance in both in-distribution and out-of-distribution scenarios. We further propose a stepwise tree search guided by a process reward model, which notably enhances robustness in out-of-distribution conditions.