SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent

TL;DR

SkyRL-Agent is a versatile framework that enhances multi-turn RL training efficiency for LLM agents, enabling faster training, better code navigation, and broad task generalization.

Contribution

It introduces an optimized asynchronous pipeline and tool-enhanced training recipe, significantly improving RL training speed and effectiveness for multi-task LLM agents.

Findings

Achieved 1.55x speedup with the pipeline dispatcher.

SA-SWE-32B reaches 39.4% Pass@1 on SWE-Bench.

Model generalizes to various agentic tasks.

Abstract

We introduce SkyRL-Agent, a framework for efficient, multi-turn, long-horizon agent training and evaluation. It provides efficient asynchronous dispatching, lightweight tool integration, and flexible backend interoperability, enabling seamless use with existing RL frameworks such as SkyRL-train, VeRL, and Tinker. Using SkyRL-Agent, we train SA-SWE-32B, a software engineering agent trained from Qwen3-32B (24.4% Pass@1) purely with reinforcement learning. We introduce two key components: an optimized asynchronous pipeline dispatcher that achieves a 1.55x speedup over naive asynchronous batching, and a tool-enhanced training recipe leveraging an AST-based search tool to facilitate code navigation, boost rollout Pass@K, and improve training efficiency. Together, these optimizations enable SA-SWE-32B to reach 39.4% Pass@1 on SWE-Bench Verified with more than 2x cost reduction compared to prior models reaching similar performance. Despite being trained solely on SWE tasks, SA-SWE-32B generalizes effectively to other agentic tasks, including Terminal-Bench, BrowseComp-Plus, and WebArena. We further demonstrate SkyRL-Agent's extensibility through case studies on deep research, computer use, and memory agents, each trained using a different training backend.