ModelNet40-E: An Uncertainty-Aware Benchmark for Point Cloud Classification

TL;DR

ModelNet40-E is a new benchmark for evaluating point cloud classification models' robustness and uncertainty calibration under synthetic LiDAR-like noise, providing both noisy data and uncertainty annotations.

Contribution

It introduces ModelNet40-E, the first benchmark with noise and uncertainty annotations for assessing robustness and calibration of point cloud classifiers.

Findings

Point Transformer v3 shows superior calibration under noise.

All models' accuracy degrades with increased noise.

Uncertainty predictions correlate with measurement noise.

Abstract

We introduce ModelNet40-E, a new benchmark designed to assess the robustness and calibration of point cloud classification models under synthetic LiDAR-like noise. Unlike existing benchmarks, ModelNet40-E provides both noise-corrupted point clouds and point-wise uncertainty annotations via Gaussian noise parameters ({\sigma}, {\mu}), enabling fine-grained evaluation of uncertainty modeling. We evaluate three popular models-PointNet, DGCNN, and Point Transformer v3-across multiple noise levels using classification accuracy, calibration metrics, and uncertainty-awareness. While all models degrade under increasing noise, Point Transformer v3 demonstrates superior calibration, with predicted uncertainties more closely aligned with the underlying measurement uncertainty.