Segment Anything Model is a Good Teacher for Local Feature Learning

TL;DR

This paper introduces SAMFeat, a novel local feature learning framework guided by the Segment Anything Model (SAM), which enhances feature description and detection through semantic relation distillation, semantic grouping, and edge attention, achieving superior results in image matching and localization.

Contribution

The paper presents SAMFeat, a new approach that leverages SAM as a teacher for local feature learning, introducing techniques like ASRD, WSC, and EAG to improve performance on limited datasets.

Findings

Outperforms previous local features on image matching tasks.

Achieves superior results in long-term visual localization.

Demonstrates effective semantic-guided feature learning.

Abstract

Local feature detection and description play an important role in many computer vision tasks, which are designed to detect and describe keypoints in "any scene" and "any downstream task". Data-driven local feature learning methods need to rely on pixel-level correspondence for training, which is challenging to acquire at scale, thus hindering further improvements in performance. In this paper, we propose SAMFeat to introduce SAM (segment anything model), a fundamental model trained on 11 million images, as a teacher to guide local feature learning and thus inspire higher performance on limited datasets. To do so, first, we construct an auxiliary task of Attention-weighted Semantic Relation Distillation (ASRD), which distillates feature relations with category-agnostic semantic information learned by the SAM encoder into a local feature learning network, to improve local feature description using semantic discrimination. Second, we develop a technique called Weakly Supervised Contrastive Learning Based on Semantic Grouping (WSC), which utilizes semantic groupings derived from SAM as weakly supervised signals, to optimize the metric space of local descriptors. Third, we design an Edge Attention Guidance (EAG) to further improve the accuracy of local feature detection and description by prompting the network to pay more attention to the edge region guided by SAM. SAMFeat's performance on various tasks such as image matching on HPatches, and long-term visual localization on Aachen Day-Night showcases its superiority over previous local features. The release code is available at https://github.com/vignywang/SAMFeat.