nuScenes Revisited: Progress and Challenges in Autonomous Driving

TL;DR

This paper provides a comprehensive review of the nuScenes dataset, highlighting its unique features, technical details, influence on the field, and its role in advancing autonomous driving research across perception, localization, and planning.

Contribution

It offers an in-depth analysis of nuScenes' creation, extensions, and impact, along with a survey of related autonomous driving methodologies and benchmarks.

Findings

nuScenes was the first dataset with radar data and diverse urban scenes

It influenced subsequent datasets and set community standards

The paper reviews major methodological advances using nuScenes

Abstract

Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) have been revolutionized by Deep Learning. As a data-driven approach, Deep Learning relies on vast amounts of driving data, typically labeled in great detail. As a result, datasets, alongside hardware and algorithms, are foundational building blocks for the development of AVs. In this work we revisit one of the most widely used autonomous driving datasets: the nuScenes dataset. nuScenes exemplifies key trends in AV development, being the first dataset to include radar data, to feature diverse urban driving scenes from two continents, and to be collected using a fully autonomous vehicle operating on public roads, while also promoting multi-modal sensor fusion, standardized benchmarks, and a broad range of tasks including perception, localization & mapping, prediction and planning. We provide an unprecedented look into the creation of nuScenes, as well as its extensions nuImages and Panoptic nuScenes, summarizing many technical details that have hitherto not been revealed in academic publications. Furthermore, we trace how the influence of nuScenes impacted a large number of other datasets that were released later and how it defined numerous standards that are used by the community to this day. Finally, we present an overview of both official and unofficial tasks using the nuScenes dataset and review major methodological developments, thereby offering a comprehensive survey of the autonomous driving literature, with a particular focus on nuScenes.