PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics

TL;DR

This paper introduces PBEBench, a novel benchmark inspired by historical linguistics that evaluates the inductive reasoning capabilities of large language models through a multi-step programming by examples task, highlighting current models' limitations.

Contribution

It presents a scalable, automated pipeline for generating reasoning problems and two benchmark datasets, advancing the evaluation of LLMs' reasoning skills beyond domain-specific tasks.

Findings

Models with test-time compute or LCoT reasoning outperform others.

Recent models solve less than 5% of hard instances.

Scaling strategies and hyperparameters significantly affect difficulty.

Abstract

Although many benchmarks evaluate the reasoning abilities of Large Language Models (LLMs) within domains such as mathematics, coding, or data wrangling, few abstract away from domain specifics to examine reasoning as a capability in and of itself. We contribute a novel type of benchmark evaluating the inductive reasoning capabilities of LLMs that is inspired by the forward reconstruction task from historical linguistics but is formulated in an extremely simple, general way (in the form of Programming by Examples). The task involves generating a cascade of simple string rewrite programs to transform a given list of input strings into a list of desired output strings. We present a fully automated pipeline that programmatically generates problems of this type with controllable difficulty, enabling scalable evaluation of reasoning models while avoiding contamination. Using this approach, we construct two benchmarks: PBEBench-Lite, which efficiently stratifies models of varying capabilities, and PBEBench, which requires models to induce programs similar in complexity to those constructed by historical linguists. Our experiments reveal a substantial performance gap between models that leverage test-time compute or LCoT (long chain-of-thought) reasoning and those that do not. Moreover, although recent models show promise, the solve rate for both of them drops below 5% for hard instances of the PBEBench dataset (ground truth cascade lengths of 20 and 30, respectively), falling well short of realistic historical linguistics requirements even with computationally expensive, popular scaling techniques from the PBE and reasoning literature. Additionally, we also study the effectiveness of different scaling strategies and the impact of various hyperparameters on the difficulty of the generated data using gpt-oss-120b, the best-performing open-source model.