Learning Rollout from Sampling:An R1-Style Tokenized Traffic Simulation Model

TL;DR

This paper introduces R1Sim, a novel reinforcement learning-based traffic simulation model that uses entropy-guided sampling and safety-aware optimization to produce realistic, diverse, and safe multi-agent driving behaviors.

Contribution

The paper proposes a new tokenized traffic simulation policy leveraging entropy patterns for exploration and introduces Group Relative Policy Optimization for behavior refinement.

Findings

R1Sim achieves competitive performance on the Waymo Sim Agent benchmark.

Entropy-guided sampling improves exploration of high-uncertainty motion tokens.

The method produces realistic, safe, and diverse multi-agent traffic behaviors.

Abstract

Learning diverse and high-fidelity traffic simulations from human driving demonstrations is crucial for autonomous driving evaluation. The recent next-token prediction (NTP) paradigm, widely adopted in large language models (LLMs), has been applied to traffic simulation and achieves iterative improvements via supervised fine-tuning (SFT). However, such methods limit active exploration of potentially valuable motion tokens, particularly in suboptimal regions. Entropy patterns provide a promising perspective for enabling exploration driven by motion token uncertainty. Motivated by this insight, we propose a novel tokenized traffic simulation policy, R1Sim, which represents an initial attempt to explore reinforcement learning based on motion token entropy patterns, and systematically analyzes the impact of different motion tokens on simulation outcomes. Specifically, we introduce an entropy-guided adaptive sampling mechanism that focuses on previously overlooked motion tokens with high uncertainty yet high potential. We further optimize motion behaviors using Group Relative Policy Optimization (GRPO), guided by a safety-aware reward design. Overall, these components enable a balanced exploration-exploitation trade-off through diverse high-uncertainty sampling and group-wise comparative estimation, resulting in realistic, safe, and diverse multi-agent behaviors. Extensive experiments on the Waymo Sim Agent benchmark demonstrate that R1Sim achieves competitive performance compared to state-of-the-art methods.