RealEngine: Simulating Autonomous Driving in Realistic Context

TL;DR

RealEngine is a comprehensive driving simulation framework that combines advanced scene reconstruction and view synthesis to provide realistic, diverse, and scalable environments for developing and evaluating autonomous driving agents.

Contribution

It introduces a novel simulation framework that integrates 3D scene reconstruction and view synthesis for photorealistic, multi-modal, and flexible driving scenarios, addressing limitations of existing simulators.

Findings

Supports diverse traffic scenarios with high realism

Enables multi-modal sensor simulation including camera and LiDAR

Provides a reliable benchmark for autonomous driving evaluation

Abstract

Driving simulation plays a crucial role in developing reliable driving agents by providing controlled, evaluative environments. To enable meaningful assessments, a high-quality driving simulator must satisfy several key requirements: multi-modal sensing capabilities (e.g., camera and LiDAR) with realistic scene rendering to minimize observational discrepancies; closed-loop evaluation to support free-form trajectory behaviors; highly diverse traffic scenarios for thorough evaluation; multi-agent cooperation to capture interaction dynamics; and high computational efficiency to ensure affordability and scalability. However, existing simulators and benchmarks fail to comprehensively meet these fundamental criteria. To bridge this gap, this paper introduces RealEngine, a novel driving simulation framework that holistically integrates 3D scene reconstruction and novel view synthesis techniques to achieve realistic and flexible closed-loop simulation in the driving context. By leveraging real-world multi-modal sensor data, RealEngine reconstructs background scenes and foreground traffic participants separately, allowing for highly diverse and realistic traffic scenarios through flexible scene composition. This synergistic fusion of scene reconstruction and view synthesis enables photorealistic rendering across multiple sensor modalities, ensuring both perceptual fidelity and geometric accuracy. Building upon this environment, RealEngine supports three essential driving simulation categories: non-reactive simulation, safety testing, and multi-agent interaction, collectively forming a reliable and comprehensive benchmark for evaluating the real-world performance of driving agents.