RealDriveSim: A Realistic Multi-Modal Multi-Task Synthetic Dataset for Autonomous Driving

TL;DR

RealDriveSim is a comprehensive synthetic dataset for autonomous driving that offers multi-modal data and detailed annotations, enabling improved model training across various perception tasks with reduced annotation costs.

Contribution

It introduces a realistic, multi-modal synthetic dataset supporting multiple tasks and classes, filling gaps in existing datasets for autonomous driving research.

Findings

Achieves state-of-the-art results on multiple perception tasks

Supports 2D and LiDAR data with detailed annotations

Demonstrates broad applicability across domains

Abstract

As perception models continue to develop, the need for large-scale datasets increases. However, data annotation remains far too expensive to effectively scale and meet the demand. Synthetic datasets provide a solution to boost model performance with substantially reduced costs. However, current synthetic datasets remain limited in their scope, realism, and are designed for specific tasks and applications. In this work, we present RealDriveSim, a realistic multi-modal synthetic dataset for autonomous driving that not only supports popular 2D computer vision applications but also their LiDAR counterparts, providing fine-grained annotations for up to 64 classes. We extensively evaluate our dataset for a wide range of applications and domains, demonstrating state-of-the-art results compared to existing synthetic benchmarks. The dataset is publicly available at https://realdrivesim.github.io/.