OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving

TL;DR

OmniHD-Scenes is a comprehensive multimodal dataset for autonomous driving, combining high-definition sensor data from LiDAR, cameras, and radar, with detailed annotations and benchmarks to advance perception algorithms.

Contribution

The paper introduces OmniHD-Scenes, a large-scale multimodal dataset with novel 4D annotations and an automated dense occupancy ground truth pipeline for autonomous driving research.

Findings

Effective low-cost sensor configuration demonstrated

Robustness under adverse conditions validated

New benchmarks for 3D detection and semantic occupancy

Abstract

The rapid advancement of deep learning has intensified the need for comprehensive data for use by autonomous driving algorithms. High-quality datasets are crucial for the development of effective data-driven autonomous driving solutions. Next-generation autonomous driving datasets must be multimodal, incorporating data from advanced sensors that feature extensive data coverage, detailed annotations, and diverse scene representation. To address this need, we present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data. The OmniHD-Scenes dataset combines data from 128-beam LiDAR, six cameras, and six 4D imaging radar systems to achieve full environmental perception. The dataset comprises 1501 clips, each approximately 30-s long, totaling more than 450K synchronized frames and more than 5.85 million synchronized sensor data points. We also propose a novel 4D annotation pipeline. To date, we have annotated 200 clips with more than 514K precise 3D bounding boxes. These clips also include semantic segmentation annotations for static scene elements. Additionally, we introduce a novel automated pipeline for generation of the dense occupancy ground truth, which effectively leverages information from non-key frames. Alongside the proposed dataset, we establish comprehensive evaluation metrics, baseline models, and benchmarks for 3D detection and semantic occupancy prediction. These benchmarks utilize surround-view cameras and 4D imaging radar to explore cost-effective sensor solutions for autonomous driving applications. Extensive experiments demonstrate the effectiveness of our low-cost sensor configuration and its robustness under adverse conditions. Data will be released at https://www.2077ai.com/OmniHD-Scenes.