Failure Modes for Deep Learning-Based Online Mapping: How to Measure and Address Them

TL;DR

This paper introduces a framework for identifying, measuring, and addressing failure modes in deep learning-based online mapping for autonomous driving, emphasizing dataset biases and geometric diversity to improve model generalization.

Contribution

It proposes novel failure-mode metrics, dataset diagnostics, and a sparsification strategy to enhance the robustness and generalization of online mapping models.

Findings

Models' performance drops significantly with geometric novelty.

Dataset diversity improves model generalization.

Sparsification reduces redundancy and enhances performance.

Abstract

Deep learning-based online mapping has emerged as a cornerstone of autonomous driving, yet these models frequently fail to generalize beyond familiar environments. We propose a framework to identify and measure the underlying failure modes by disentangling two effects: Memorization of input features and overfitting to known map geometries. We propose measures based on evaluation subsets that control for geographical proximity and geometric similarity between training and validation scenes. We introduce Fr\'echet distance-based reconstruction statistics that capture per-element shape fidelity without threshold tuning, and define complementary failure-mode scores: a localization overfitting score quantifying the performance drop when geographic cues disappear, and a map geometry overfitting score measuring degradation as scenes become geometrically novel. Beyond models, we analyze dataset biases and contribute map geometry-aware diagnostics: A minimum-spanning-tree (MST) diversity measure for training sets and a symmetric coverage measure to quantify geometric similarity between splits. Leveraging these, we formulate an MST-based sparsification strategy that reduces redundancy and improves balancing and performance while shrinking training size. Experiments on nuScenes and Argoverse 2 across multiple state-of-the-art models yield more trustworthy assessment of generalization and show that map geometry-diverse and balanced training sets lead to improved performance. Our results motivate failure-mode-aware protocols and map geometry-centric dataset design for deployable online mapping.