Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future

TL;DR

This paper systematically reviews open-source autonomous driving datasets, highlighting their evolution from perception-focused to multi-task datasets, and discusses future directions, challenges, and the role of data generation technologies in advancing autonomous driving.

Contribution

It provides a comprehensive assessment of over seventy datasets, analyzes their characteristics, and explores future dataset development and technical challenges in autonomous driving.

Findings

First-generation datasets are perception-focused with limited sensor modalities.

Second-generation datasets feature increased complexity and task diversity.

Future datasets should incorporate higher data scales and multi-modal sensors.

Abstract

With the continuous maturation and application of autonomous driving technology, a systematic examination of open-source autonomous driving datasets becomes instrumental in fostering the robust evolution of the industry ecosystem. Current autonomous driving datasets can broadly be categorized into two generations. The first-generation autonomous driving datasets are characterized by relatively simpler sensor modalities, smaller data scale, and is limited to perception-level tasks. KITTI, introduced in 2012, serves as a prominent representative of this initial wave. In contrast, the second-generation datasets exhibit heightened complexity in sensor modalities, greater data scale and diversity, and an expansion of tasks from perception to encompass prediction and control. Leading examples of the second generation include nuScenes and Waymo, introduced around 2019. This comprehensive review, conducted in collaboration with esteemed colleagues from both academia and industry, systematically assesses over seventy open-source autonomous driving datasets from domestic and international sources. It offers insights into various aspects, such as the principles underlying the creation of high-quality datasets, the pivotal role of data engine systems, and the utilization of generative foundation models to facilitate scalable data generation. Furthermore, this review undertakes an exhaustive analysis and discourse regarding the characteristics and data scales that future third-generation autonomous driving datasets should possess. It also delves into the scientific and technical challenges that warrant resolution. These endeavors are pivotal in advancing autonomous innovation and fostering technological enhancement in critical domains. For further details, please refer to https://github.com/OpenDriveLab/DriveAGI.