Pseudo-Simulation for Autonomous Driving

TL;DR

Pseudo-simulation offers a new evaluation paradigm for autonomous vehicles that combines the efficiency of open-loop methods with the realism of closed-loop simulation by augmenting real datasets with synthetic observations generated through 3D Gaussian Splatting.

Contribution

This paper introduces pseudo-simulation, a novel evaluation approach that enhances open-loop datasets with synthetic, diverse observations to better approximate real-world AV scenarios.

Findings

Pseudo-simulation correlates more strongly with closed-loop simulations ($R^2=0.8$) than existing open-loop methods ($R^2=0.7$).

The method enables error recovery and causal confusion mitigation without interactive simulation.

A public leaderboard for benchmarking pseudo-simulation is established.

Abstract

Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations. Real-world evaluation is often challenging due to safety concerns and a lack of reproducibility, whereas closed-loop simulation can face insufficient realism or high computational costs. Open-loop evaluation, while being efficient and data-driven, relies on metrics that generally overlook compounding errors. In this paper, we propose pseudo-simulation, a novel paradigm that addresses these limitations. Pseudo-simulation operates on real datasets, similar to open-loop evaluation, but augments them with synthetic observations generated prior to evaluation using 3D Gaussian Splatting. Our key idea is to approximate potential future states the AV might encounter by generating a diverse set of observations that vary in position, heading, and speed. Our method then assigns a higher importance to synthetic observations that best match the AV's likely behavior using a novel proximity-based weighting scheme. This enables evaluating error recovery and the mitigation of causal confusion, as in closed-loop benchmarks, without requiring sequential interactive simulation. We show that pseudo-simulation is better correlated with closed-loop simulations ($R^2=0.8$) than the best existing open-loop approach ($R^2=0.7$). We also establish a public leaderboard for the community to benchmark new methodologies with pseudo-simulation. Our code is available at https://github.com/autonomousvision/navsim.