Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations

TL;DR

This paper empirically evaluates how large language models handle various structured perturbations in chain-of-thought reasoning, revealing different vulnerability patterns and the impact of model scaling.

Contribution

It provides a comprehensive analysis of LLM robustness to five types of reasoning perturbations across multiple model sizes, highlighting scaling effects and robustness challenges.

Findings

MathError perturbations cause significant accuracy loss in small models but improve with scale.

UnitConversion remains difficult across all model sizes.

ExtraSteps perturbations minimally affect accuracy even in small models.

Abstract

Chain-of-Thought (CoT) prompting has emerged as a foundational technique for eliciting reasoning from Large Language Models (LLMs), yet the robustness of this approach to corruptions in intermediate reasoning steps remains poorly understood. This paper presents a comprehensive empirical evaluation of LLM robustness to a structured taxonomy of 5 CoT perturbation types: \textit{MathError, UnitConversion, Sycophancy, SkippedSteps,} and \textit{ExtraSteps}. We evaluate 13 models spanning three orders of magnitude in parameter count, testing their ability to complete mathematical reasoning tasks despite perturbations injected in the reasoning chain. Our key findings reveal heterogeneous vulnerability patterns: MathError perturbations produce the most severe degradation in small models (50-60\% accuracy loss) but show strong scaling benefits; UnitConversion remains challenging across all scales (>5\% loss even for midsized models); ExtraSteps incur minimal accuracy degradation (0-6\%) even for the smallest of models; Sycophancy and SkippedSteps produce modest effects ($\sim$10\% loss for small models) and slightly improve with scale. Scaling relationships show that model size serve as a protective factor against many perturbations but not always. These findings have direct implications for deploying LLMs in multi-stage reasoning pipelines and underscore the necessity of task-specific robustness assessments and mitigation strategies. The code and results are available at https://github.com/Mystic-Slice/CoTPerturbation