Adaptive Problem Generation via Symbolic Representations

TL;DR

This paper introduces a symbolic, adaptive problem generation method for reinforcement learning that enhances small language models' mathematical reasoning by controlling problem structure and difficulty.

Contribution

It proposes a symbolic problem space for precise control and an adaptive framework that learns to modify problems, improving mathematical reasoning in language models.

Findings

Adaptive problem generation improves model performance.

Symbolic representations enable diverse and controlled problem creation.

Learning modification strategies enhances math-solving ability.

Abstract

We present a method for generating training data for reinforcement learning with verifiable rewards to improve small open-weights language models on mathematical tasks. Existing data generation approaches rely on open-loop pipelines and fixed modifications that do not adapt to the model's capabilities. Furthermore, they typically operate directly on word problems, limiting control over problem structure. To address this, we perform modifications in a symbolic problem space, representing each problem as a set of symbolic variables and constraints (e.g., via algebraic frameworks such as SymPy or SMT formulations). This representation enables precise control over problem structure, automatic generation of ground-truth solutions, and decouples mathematical reasoning from linguistic realization. We also show that this results in more diverse generations. To adapt the problem difficulty to the model, we introduce a closed-loop framework that learns modification strategies through prompt optimization in symbolic space. Experimental results demonstrate that both adaptive problem generation and symbolic representation modifications contribute to improving the model's math solving ability.