Endless Terminals: Scaling RL Environments for Terminal Agents

TL;DR

This paper introduces Endless Terminals, an autonomous pipeline for generating diverse terminal-use tasks to train reinforcement learning agents, resulting in significant performance improvements on various benchmarks with simple RL methods.

Contribution

The paper presents a scalable, automated environment generation pipeline for RL training, enabling effective learning with minimal interaction and simple algorithms.

Findings

Agents trained on Endless Terminals outperform baselines on held-out benchmarks.

Models show substantial gains in task success rates after training.

Simple RL methods benefit greatly from the scalable environment pipeline.

Abstract

Environments are the bottleneck for self-improving agents. Current terminal benchmarks were built for evaluation, not training; reinforcement learning requires a scalable pipeline, not just a dataset. We introduce Endless Terminals, a fully autonomous pipeline that procedurally generates terminal-use tasks without human annotation. The pipeline has four stages: generating diverse task descriptions, building and validating containerized environments, producing completion tests, and filtering for solvability. From this pipeline we obtain 3255 tasks spanning file operations, log management, data processing, scripting, and database operations. We train agents using vanilla PPO with binary episode level rewards and a minimal interaction loop: no retrieval, multi-agent coordination, or specialized tools. Despite this simplicity, models trained on Endless Terminals show substantial gains: on our held-out dev set, Llama-3.2-3B improves from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%. These improvements transfer to human-curated benchmarks: models trained on Endless Terminals show substantial gains on held out human curated benchmarks: on TerminalBench 2.0, Llama-3.2-3B improves from 0.0% to 2.2%, Qwen2.5-7B from 2.2% to 3.4%, and Qwen3-8B-openthinker-sft from 1.1% to 6.7%, in each case outperforming alternative approaches including models with more complex agentic scaffolds. These results demonstrate that simple RL succeeds when environments scale.