StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models

TL;DR

StressEval is a framework that creates dynamic, challenging, and controllable test instances for large language models by transforming observed failures into new benchmark data.

Contribution

It introduces a failure-driven data synthesis method that enhances knowledge-intensive reasoning benchmarks with controllable difficulty and grounded instances.

Findings

Dynamic OneEval causes larger performance drops than static benchmarks.

StressEval retains explicit difficulty factors for actionable model improvement.

The framework effectively targets knowledge gaps and reasoning breakdowns.

Abstract

Static benchmarks for LLMs are increasingly compromised by contamination and overfitting especially on knowledge intensive reasoning tasks While recent dynamic benchmarks can alleviate staleness they often increase difficulty at the expense of answerability and controllability In this paper we propose StressEval a failure driven data synthesis framework that turns observed model failures into dynamic challenging and controllable test instances StressEval consists of three stages first it constructs a semi structured difficulty card that identifies the failed reasoning step and its root cause second it applies a dual perspective instance synthesis method that targets both knowledge gaps and reasoning breakdowns while preserving the underlying difficulty factors and third it applies a gating mechanism to retain only grounded unambiguous instances Seeding from multiple knowledge intensive reasoning datasets we employ StressEval to build Dynamic OneEval a focused suite of challenging dynamic benchmark Across several state of the art LLMs Dynamic OneEval yields substantially larger performance drops than the original benchmarks while retaining explicit difficulty factors enabling more actionable iteration