Systematic Benchmarking of SUMO Against Data-Driven Traffic Simulators

TL;DR

This paper systematically compares the traditional model-based SUMO traffic simulator with modern data-driven simulators using large real-world datasets, highlighting strengths and limitations of each approach.

Contribution

It introduces Waymo2SUMO, an automated pipeline for scalable SUMO simulation, and provides a comprehensive benchmark using real-world data to evaluate realism and stability.

Findings

SUMO achieves a realism metric of 0.653 on WOSAC

SUMO maintains low collision and offroad rates in extended simulations

SUMO exhibits stronger long-horizon stability than data-driven simulators

Abstract

This paper presents a systematic benchmarking of the model-based microscopic traffic simulator SUMO against state-of-the-art data-driven traffic simulators using large-scale real-world datasets. Using the Waymo Open Motion Dataset (WOMD) and the Waymo Open Sim Agents Challenge (WOSAC), we evaluate SUMO under both short-horizon (8s) and long-horizon (60s) closed-loop simulation settings. To enable scalable evaluation, we develop Waymo2SUMO, an automated pipeline that converts WOMD scenarios into SUMO simulations. On the WOSAC benchmark, SUMO achieves a realism meta metric of 0.653 while requiring fewer than 100 tunable parameters. Extended rollouts show that SUMO maintains low collision and offroad rates and exhibits stronger long-horizon stability than representative data-driven simulators. These results highlight complementary strengths of model-based and data-driven approaches for autonomous driving simulation and benchmarking.