Towards Long-Tailed 3D Detection

TL;DR

This paper introduces Long-Tailed 3D Detection (LT3D), a framework that evaluates and improves detection of rare classes in autonomous vehicle data by leveraging hierarchical relationships and multimodal fusion, significantly boosting accuracy especially for tail classes.

Contribution

The paper formalizes LT3D, adapting existing 3D detection models for long-tailed class distributions, and proposes hierarchical losses and multimodal fusion techniques to enhance rare class detection.

Findings

Accuracy for all classes improved by 5% AP on average.

Significant boost in rare class AP, e.g., stroller AP from 3.6 to 31.6.

Hierarchical and multimodal methods are key to long-tailed 3D detection.

Abstract

Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale lidar data. Surprisingly, although semantic class labels naturally follow a long-tailed distribution, contemporary benchmarks focus on only a few common classes (e.g., pedestrian and car) and neglect many rare classes in-the-tail (e.g., debris and stroller). However, AVs must still detect rare classes to ensure safe operation. Moreover, semantic classes are often organized within a hierarchy, e.g., tail classes such as child and construction-worker are arguably subclasses of pedestrian. However, such hierarchical relationships are often ignored, which may lead to misleading estimates of performance and missed opportunities for algorithmic innovation. We address these challenges by formally studying the problem of Long-Tailed 3D Detection (LT3D), which evaluates on all classes, including those in-the-tail. We evaluate and innovate upon popular 3D detection codebases, such as CenterPoint and PointPillars, adapting them for LT3D. We develop hierarchical losses that promote feature sharing across common-vs-rare classes, as well as improved detection metrics that award partial credit to "reasonable" mistakes respecting the hierarchy (e.g., mistaking a child for an adult). Finally, we point out that fine-grained tail class accuracy is particularly improved via multimodal fusion of RGB images with LiDAR; simply put, small fine-grained classes are challenging to identify from sparse (lidar) geometry alone, suggesting that multimodal cues are crucial to long-tailed 3D detection. Our modifications improve accuracy by 5% AP on average for all classes, and dramatically improve AP for rare classes (e.g., stroller AP improves from 3.6 to 31.6)! Our code is available at https://github.com/neeharperi/LT3D