ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models

TL;DR

ReSyn is a scalable pipeline that automatically generates diverse reasoning environments to improve the training of reasoning language models, leading to significant performance gains across multiple benchmarks.

Contribution

The paper introduces ReSyn, a novel method for large-scale synthetic environment generation for reasoning models, moving beyond solution-centric data to enhance reasoning capabilities.

Findings

Qwen2.5-7B-Instruct trained on ReSyn data outperforms baselines.

Verifier-based supervision improves reasoning performance.

Increased environment diversity enhances model reasoning abilities.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs) by leveraging supervision from verifiers. Although verifier implementation is easier than solution annotation for many tasks, existing synthetic data generation methods remain largely solution-centric, while verifier-based methods rely on a few hand-crafted procedural environments. In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers, covering tasks such as constraint satisfaction, algorithmic puzzles, and spatial reasoning. A Qwen2.5-7B-Instruct model trained with RL on ReSyn data achieves consistent gains across reasoning benchmarks and out-of-domain math benchmarks, including a 27\% relative improvement on the challenging BBEH benchmark. Ablations show that verifier-based supervision and increased task diversity both contribute significantly, providing empirical evidence that generating reasoning environments at scale can enhance reasoning abilities in RLMs