Open-Set Semantic Uncertainty Aware Metric-Semantic Graph Matching

TL;DR

This paper introduces a novel semantic uncertainty metric and a graph matching approach for robust, real-time loop closure detection in underwater and terrestrial environments, handling unseen object classes effectively.

Contribution

It proposes a new semantic uncertainty measure integrated into a graph matching framework for open-set object detection in marine and terrestrial mapping.

Findings

The method achieves real-time performance in challenging underwater scenes.

It generalizes well to large-scale terrestrial datasets like KITTI.

The proposed approach outperforms existing methods in open-set loop closure detection.

Abstract

Underwater object-level mapping requires incorporating visual foundation models to handle the uncommon and often previously unseen object classes encountered in marine scenarios. In this work, a metric of semantic uncertainty for open-set object detections produced by visual foundation models is calculated and then incorporated into an object-level uncertainty tracking framework. Object-level uncertainties and geometric relationships between objects are used to enable robust object-level loop closure detection for unknown object classes. The above loop closure detection problem is formulated as a graph-matching problem. While graph matching, in general, is NP-Complete, a solver for an equivalent formulation of the proposed graph matching problem as a graph editing problem is tested on multiple challenging underwater scenes. Results for this solver as well as three other solvers demonstrate that the proposed methods are feasible for real-time use in marine environments for the robust, open-set, multi-object, semantic-uncertainty-aware loop closure detection. Further experimental results on the KITTI dataset demonstrate that the method generalizes to large-scale terrestrial scenes.