Beyond Self-Play: Hierarchical Reasoning for Continuous Motion in Closed-Loop Traffic Simulation

TL;DR

This paper introduces a hierarchical multi-agent reinforcement learning framework for realistic and safe continuous motion control in traffic simulation, surpassing self-play methods.

Contribution

It combines high-level interaction reasoning with low-level motion control, using a hybrid co-training scheme to improve closed-loop deployment performance.

Findings

Achieves smoother and safer traffic control in urban simulations.

Outperforms self-play and imitation baselines in safety and control quality.

Maintains competitive traffic efficiency.

Abstract

Closed-loop traffic simulation requires agents that are both scalable and behaviorally realistic. Recent self-play reinforcement learning approaches demonstrate strong scalability, but their equilibrium strategies fail to capture the socially aware behaviors of real human drivers. We propose a hierarchical architecture that goes beyond self-play by combining high-level multi-agent interaction reasoning with low-level continuous trajectory realization. Specifically, a Stackelberg-style Multi-Agent Reinforcement Learning (MARL) module generates interaction-aware intention commands. These commands condition a low-level continuous motion module, translating the strategic intent into physically consistent, scene-responsive control sequences. To mitigate distribution shift in closed-loop deployment, we introduce a hybrid co-training scheme combining MARL with auxiliary recovery supervision. Experiments on a SUMO-based urban network demonstrate that the proposed framework achieves superior control smoothness and safety compared to self-play and passive imitation baselines, while maintaining competitive traffic efficiency.