HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving

TL;DR

HybridWorldSim is a novel high-fidelity simulation framework for autonomous driving that combines neural reconstruction and generative modeling to produce diverse, realistic, and consistent driving scenarios, addressing key limitations of existing simulators.

Contribution

It introduces HybridWorldSim, a hybrid simulation system integrating multi-traversal neural reconstruction with generative modeling, and releases MIRROR, a comprehensive dataset for benchmarking.

Findings

HybridWorldSim outperforms previous methods in realism and consistency.

The framework enables scalable and controllable scenario generation.

Extensive experiments validate its effectiveness for autonomous driving research.

Abstract

Realistic and controllable simulation is critical for advancing end-to-end autonomous driving, yet existing approaches often struggle to support novel view synthesis under large viewpoint changes or to ensure geometric consistency. We introduce HybridWorldSim, a hybrid simulation framework that integrates multi-traversal neural reconstruction for static backgrounds with generative modeling for dynamic agents. This unified design addresses key limitations of previous methods, enabling the creation of diverse and high-fidelity driving scenarios with reliable visual and spatial consistency. To facilitate robust benchmarking, we further release a new multi-traversal dataset MIRROR that captures a wide range of routes and environmental conditions across different cities. Extensive experiments demonstrate that HybridWorldSim surpasses previous state-of-the-art methods, providing a practical and scalable solution for high-fidelity simulation and a valuable resource for research and development in autonomous driving.