TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization

TL;DR

TL-GRPO is a novel turn-level reinforcement learning algorithm designed for iterative reasoning tasks, outperforming existing methods in complex scientific optimization like analog circuit sizing.

Contribution

We introduce TL-GRPO, a lightweight turn-level RL method that enables fine-grained optimization in reasoning tasks with shared environment states, addressing limitations of previous trajectory-level approaches.

Findings

TL-GRPO outperforms standard GRPO and Bayesian optimization in analog circuit sizing.

A 30B model trained with TL-GRPO achieves state-of-the-art results.

TL-GRPO demonstrates strong generalization and practical utility in scientific optimization.

Abstract

Large language models have demonstrated strong reasoning capabilities in complex tasks through tool integration, which is typically framed as a Markov Decision Process and optimized with trajectory-level RL algorithms such as GRPO. However, a common class of reasoning tasks, iterative optimization, presents distinct challenges: the agent interacts with the same underlying environment state across turns, and the value of a trajectory is determined by the best turn-level reward rather than cumulative returns. Existing GRPO-based methods cannot perform fine-grained, turn-level optimization in such settings, while black-box optimization methods discard prior knowledge and reasoning capabilities. To address this gap, we propose Turn-Level GRPO (TL-GRPO), a lightweight RL algorithm that performs turn-level group sampling for fine-grained optimization. We evaluate TL-GRPO on analog circuit sizing (ACS), a challenging scientific optimization task requiring multiple simulations and domain expertise. Results show that TL-GRPO outperforms standard GRPO and Bayesian optimization methods across various specifications. Furthermore, our 30B model trained with TL-GRPO achieves state-of-the-art performance on ACS tasks under same simulation budget, demonstrating both strong generalization and practical utility.