Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentation

TL;DR

This paper proposes a novel label-free failure detection method for lidar point cloud segmentation in autonomous vehicles, utilizing unlabeled data and dual training streams, and introduces a new dataset with labeled anomalies for evaluation.

Contribution

It introduces a new label-free failure detection approach using supervised and self-supervised streams and provides the first dataset with labeled anomalies in real-world lidar data.

Findings

Effective failure detection on unlabeled data

Large-scale qualitative analysis conducted

Introduction of LidarCODA dataset with labeled anomalies

Abstract

Autonomous vehicles drive millions of miles on the road each year. Under such circumstances, deployed machine learning models are prone to failure both in seemingly normal situations and in the presence of outliers. However, in the training phase, they are only evaluated on small validation and test sets, which are unable to reveal model failures due to their limited scenario coverage. While it is difficult and expensive to acquire large and representative labeled datasets for evaluation, large-scale unlabeled datasets are typically available. In this work, we introduce label-free model failure detection for lidar-based point cloud segmentation, taking advantage of the abundance of unlabeled data available. We leverage different data characteristics by training a supervised and self-supervised stream for the same task to detect failure modes. We perform a large-scale qualitative analysis and present LidarCODA, the first publicly available dataset with labeled anomalies in real-world lidar data, for an extensive quantitative analysis.