Semantic SuperPoint: A Deep Semantic Descriptor

TL;DR

Semantic SuperPoint introduces a multi-task learning approach with a shared encoder and semantic segmentation decoder to enhance feature descriptors for SLAM, leading to improved detection and matching performance.

Contribution

It proposes a novel multi-task learning architecture that integrates semantic segmentation into feature descriptor learning for more robust SLAM features.

Findings

Semantic SuperPoint outperforms baseline in detection metrics.

Multi-task learning improves descriptor robustness.

Semantic information enhances feature matching accuracy.

Abstract

Several SLAM methods benefit from the use of semantic information. Most integrate photometric methods with high-level semantics such as object detection and semantic segmentation. We propose that adding a semantic segmentation decoder in a shared encoder architecture would help the descriptor decoder learn semantic information, improving the feature extractor. This would be a more robust approach than only using high-level semantic information since it would be intrinsically learned in the descriptor and would not depend on the final quality of the semantic prediction. To add this information, we take advantage of multi-task learning methods to improve accuracy and balance the performance of each task. The proposed models are evaluated according to detection and matching metrics on the HPatches dataset. The results show that the Semantic SuperPoint model performs better than the baseline one.