SLR: Automated Synthesis for Scalable Logical Reasoning

TL;DR

SLR is an automated framework for evaluating and training large language models in logical reasoning, creating a comprehensive benchmark and demonstrating improved performance with curriculum learning.

Contribution

Introduces SLR, an automated, scalable method for synthesizing reasoning tasks and a large benchmark, enabling effective training and evaluation of LLMs in logical reasoning.

Findings

Contemporary LLMs produce valid rules but often fail in logical inference.

Recent reasoning LLMs improve accuracy but have high computational costs.

Curriculum learning with SLR doubles Llama-3-8B accuracy on SLR-Bench.

Abstract

We introduce SLR, an end-to-end framework for systematic evaluation and training of Large Language Models (LLMs) via Scalable Logical Reasoning. Given a user's task specification, SLR automatically synthesizes (i) an instruction prompt for an inductive reasoning task, (ii) a validation program, executable on model outputs to provide verifiable rewards, and (iii) the latent ground-truth rule. This process is fully automated, scalable, requires no human annotations, and offers precise control over task difficulty. Using SLR, we create SLR-Bench, a benchmark comprising 19k prompts organized into 20 curriculum levels that progressively increase in relational, arithmetic, and recursive complexity. Large-scale evaluation reveals that contemporary LLMs readily produce syntactically valid rules, yet often fail at correct logical inference. Recent reasoning LLMs demonstrate improved performance but incur very high test-time computation, with costs exceeding $300 for just 1,000 prompts. Finally, curriculum learning via SLR doubles Llama-3-8B accuracy on SLR-Bench, achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. Moreover, these reasoning capabilities generalize to a wide range of established benchmarks, underscoring the effectiveness of SLR for downstream reasoning.