LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning framework that fine-tunes large language models using logic puzzles to enhance their general reasoning abilities and improve performance on diverse mathematical tasks.

Contribution

It presents a novel 'play to learn' approach that uses custom logic puzzles and reinforcement learning to cultivate transferable reasoning skills in LLMs, surpassing traditional fine-tuning methods.

Findings

Improved out-of-distribution performance on mathematical benchmarks.

Enhanced reasoning in algebra, geometry, and combinatorics.

Limited gains on rote or highly specialized tasks.

Abstract

Large language models (LLMs) excel at many supervised tasks but often struggle with structured reasoning in unfamiliar settings. This discrepancy suggests that standard fine-tuning pipelines may instill narrow, domain-specific heuristics rather than fostering general-purpose thinking strategies. In this work, we propose a "play to learn" framework that fine-tunes LLMs through reinforcement learning on a suite of seven custom logic puzzles, each designed to cultivate distinct reasoning skills such as constraint propagation, spatial consistency, and symbolic deduction. Using a reinforcement learning setup with verifiable rewards, models receive binary feedback based on puzzle correctness, encouraging iterative, hypothesis-driven problem solving. We demonstrate that this training approach significantly improves out-of-distribution performance on a range of mathematical benchmarks, especially for mid-difficulty problems that require multi-step reasoning. Analyses across problem categories and difficulty levels reveal that puzzle training promotes transferable reasoning routines, strengthening algebraic manipulation, geometric inference, and combinatorial logic, while offering limited gains on rote or highly specialized tasks. These findings show that reinforcement learning over logic puzzles reshapes the internal reasoning of LLMs, enabling more robust and compositional generalization without relying on task-specific symbolic tools.