VERIFY-RL: Verifiable Recursive Decomposition for Reinforcement Learning in Mathematical Reasoning

TL;DR

Verify-RL introduces a mathematically grounded, verifiable decomposition framework for reinforcement learning in mathematical reasoning, significantly improving accuracy by ensuring valid problem breakdowns.

Contribution

It presents a novel verification-based decomposition method using symbolic differentiation, ensuring valid subproblem generation with provable properties, unlike heuristic approaches.

Findings

Accuracy on hardest problems more than doubles from 32% to 68%.

Eliminating invalid decompositions improves overall performance.

Framework provides automatic verification through symbolic computation.

Abstract

Training language models to solve complex mathematical problems benefits from curriculum learning progressively training on simpler subproblems. However, existing decomposition methods are often heuristic, offering no guarantees that subproblems are simpler, that solving them aids the parent task, or that their relationships are mathematically grounded. We observe that symbolic differentiation provides a natural structure for verified decomposition: calculus rules explicitly define how expressions reduce to simpler components with provable properties. We introduce Verify-RL, a framework where every parent-child decomposition satisfies three verifiable conditions: strictly decreasing structural complexity, solution containment, and formal rule derivation. Unlike heuristic methods where a significant fraction of decompositions are invalid our properties admit automatic verification through symbolic computation, achieving "verification by construction" Experiments demonstrate that eliminating invalid decompositions yields sizable gains, accuracy on the hardest problems more than doubles from 32% to 68%, with a 40% relative improvement overall.