DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation

TL;DR

DrivingSphere introduces a high-fidelity, closed-loop simulation framework that constructs detailed 4D driving worlds with realistic visual outputs, enabling better testing and validation of autonomous driving systems.

Contribution

It presents a novel 4D world modeling and visual synthesis approach for closed-loop autonomous driving simulation, addressing limitations of existing open-loop and feedback-driven environments.

Findings

Constructs detailed 4D driving worlds with static and dynamic elements.

Generates high-fidelity, multi-view visual outputs with spatial and temporal consistency.

Enables comprehensive testing of autonomous driving algorithms.

Abstract

Autonomous driving evaluation requires simulation environments that closely replicate actual road conditions, including real-world sensory data and responsive feedback loops. However, many existing simulations need to predict waypoints along fixed routes on public datasets or synthetic photorealistic data, \ie, open-loop simulation usually lacks the ability to assess dynamic decision-making. While the recent efforts of closed-loop simulation offer feedback-driven environments, they cannot process visual sensor inputs or produce outputs that differ from real-world data. To address these challenges, we propose DrivingSphere, a realistic and closed-loop simulation framework. Its core idea is to build 4D world representation and generate real-life and controllable driving scenarios. In specific, our framework includes a Dynamic Environment Composition module that constructs a detailed 4D driving world with a format of occupancy equipping with static backgrounds and dynamic objects, and a Visual Scene Synthesis module that transforms this data into high-fidelity, multi-view video outputs, ensuring spatial and temporal consistency. By providing a dynamic and realistic simulation environment, DrivingSphere enables comprehensive testing and validation of autonomous driving algorithms, ultimately advancing the development of more reliable autonomous cars. The benchmark will be publicly released.