RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case

TL;DR

RoboTron-Sim enhances autonomous driving in critical real-world situations by leveraging a new simulated dataset and multimodal language models to better handle rare, high-risk scenarios, achieving significant performance improvements.

Contribution

The paper introduces HASS, a comprehensive simulated dataset for high-risk scenarios, and novel methods SPE and I2E Encoder for effective transfer learning to real-world driving tasks.

Findings

Improves driving performance in challenging scenarios by around 50%.

Achieves state-of-the-art results in real-world open-loop planning.

Effectively manages rare high-risk driving scenarios.

Abstract

Collecting real-world data for rare high-risk scenarios, long-tailed driving events, and complex interactions remains challenging, leading to poor performance of existing autonomous driving systems in these critical situations. In this paper, we propose RoboTron-Sim that improves real-world driving in critical situations by utilizing simulated hard cases. First, we develop a simulated dataset called Hard-case Augmented Synthetic Scenarios (HASS), which covers 13 high-risk edge-case categories, as well as balanced environmental conditions such as day/night and sunny/rainy. Second, we introduce Scenario-aware Prompt Engineering (SPE) and an Image-to-Ego Encoder (I2E Encoder) to enable multimodal large language models to effectively learn real-world challenging driving skills from HASS, via adapting to environmental deviations and hardware differences between real-world and simulated scenarios. Extensive experiments on nuScenes show that RoboTron-Sim improves driving performance in challenging scenarios by around 50%, achieving state-of-the-art results in real-world open-loop planning. Qualitative results further demonstrate the effectiveness of RoboTron-Sim in better managing rare high-risk driving scenarios. Project page: https://stars79689.github.io/RoboTron-Sim/