RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation

TL;DR

RE-IMAGINE introduces a symbolic benchmark synthesis framework inspired by the ladder of causation, enabling evaluation of LLM reasoning abilities across different levels and revealing reliance on memorization.

Contribution

The paper presents a novel framework for generating reasoning problems at multiple levels of the causation hierarchy, allowing for more accurate assessment of LLM reasoning skills.

Findings

Performance drops on problem variations indicate reliance on statistical recall.

Framework is applicable across math, code, and logic domains.

Demonstrates the need for models to develop deeper reasoning skills.

Abstract

Recent Large Language Models (LLMs) have reported high accuracy on reasoning benchmarks. However, it is still unclear whether the observed results arise from true reasoning or from statistical recall of the training set. Inspired by the ladder of causation (Pearl, 2009) and its three levels (associations, interventions and counterfactuals), this paper introduces RE-IMAGINE, a framework to characterize a hierarchy of reasoning ability in LLMs, alongside an automated pipeline to generate problem variations at different levels of the hierarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone. Moreover, the framework is general and can work across reasoning domains, including math, code, and logic. We demonstrate our framework on four widely-used benchmarks to evaluate several families of LLMs, and observe reductions in performance when the models are queried with problem variations. These assessments indicate a degree of reliance on statistical recall for past performance, and open the door to further research targeting skills across the reasoning hierarchy.