STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenes

TL;DR

STCrowd is a large-scale multimodal dataset designed to improve pedestrian detection and tracking in crowded scenes, featuring synchronized LiDAR and camera data with extensive annotations, along with a novel density-aware method for better perception.

Contribution

The paper introduces STCrowd, a comprehensive dataset for pedestrian perception in crowded scenes, and proposes DHA, a novel density-aware method to enhance detection accuracy.

Findings

DHA achieves state-of-the-art detection performance.

STCrowd contains 219K pedestrian instances with diverse occlusion levels.

Extensive experiments validate the effectiveness of the proposed method.

Abstract

Accurately detecting and tracking pedestrians in 3D space is challenging due to large variations in rotations, poses and scales. The situation becomes even worse for dense crowds with severe occlusions. However, existing benchmarks either only provide 2D annotations, or have limited 3D annotations with low-density pedestrian distribution, making it difficult to build a reliable pedestrian perception system especially in crowded scenes. To better evaluate pedestrian perception algorithms in crowded scenarios, we introduce a large-scale multimodal dataset,STCrowd. Specifically, in STCrowd, there are a total of 219 K pedestrian instances and 20 persons per frame on average, with various levels of occlusion. We provide synchronized LiDAR point clouds and camera images as well as their corresponding 3D labels and joint IDs. STCrowd can be used for various tasks, including LiDAR-only, image-only, and sensor-fusion based pedestrian detection and tracking. We provide baselines for most of the tasks. In addition, considering the property of sparse global distribution and density-varying local distribution of pedestrians, we further propose a novel method, Density-aware Hierarchical heatmap Aggregation (DHA), to enhance pedestrian perception in crowded scenes. Extensive experiments show that our new method achieves state-of-the-art performance for pedestrian detection on various datasets.