LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling

TL;DR

This paper introduces LPDS, a framework for systematically evaluating LLM robustness by identifying and testing the most challenging logic-preserving variations, revealing significant performance drops and guiding more effective fine-tuning.

Contribution

LPDS provides a systematic method to quantify and find the most difficult problem variations, improving robustness evaluation and training strategies for LLMs.

Findings

Performance drops up to 5 times larger with LPDS compared to random sampling.

LPDS efficiently finds difficult variations that induce model failures.

Fine-tuning on difficult variations yields more consistent robustness improvements.

Abstract

As large language models (LLMs) are increasingly deployed to perform tasks with minimal human oversight, it is crucial that these models operate robustly. In particular, a model that can solve a given problem should not fail simply because certain entities$\unicode{x2013}$such as names, numbers, or other contextual details$\unicode{x2013}$have changed while the underlying problem logic remains the same. Prior work suggests that current LLMs still struggle with this form of robustness: they often succeed on some variations of a problem but fail on others. However, existing evaluations often lack a systematic way to identify which logic-preserving variations are most likely to induce failure. Instead, they typically test a random subset of allowable variations, which can overstate robustness. To address this gap, we introduce logic-preserving difficulty scaling (LPDS), a framework that (i) quantifies the difficulty of a problem variation and (ii) systematically searches the space of allowable variations to find those that maximize difficulty and expose failures. We show that as difficulty increases, performance declines and errors in the models' reasoning chains become more pronounced. We further demonstrate that LPDS efficiently finds difficult problem variations for a model, resulting in performance drops up to 5 times larger compared to random sampling. Finally, we show that fine-tuning on more difficult variations leads to more consistent robustness gains than training on easier ones.