Semantic-aware Representation Learning for Homography Estimation

TL;DR

This paper introduces SRMatcher, a semantic-aware, detector-free feature matching method utilizing vision foundation models and a fusion block to improve homography estimation accuracy, surpassing state-of-the-art results.

Contribution

The paper presents a novel semantic-aware feature learning framework with a fusion block, enhancing detector-free matching methods for homography estimation.

Findings

SRMatcher achieves state-of-the-art performance on multiple datasets.

It increases the AUC by about 11% on HPatches compared to previous methods.

SRMatcher improves precision when integrated with other matching frameworks like LoFTR.

Abstract

Homography estimation is the task of determining the transformation from an image pair. Our approach focuses on employing detector-free feature matching methods to address this issue. Previous work has underscored the importance of incorporating semantic information, however there still lacks an efficient way to utilize semantic information. Previous methods suffer from treating the semantics as a pre-processing, causing the utilization of semantics overly coarse-grained and lack adaptability when dealing with different tasks. In our work, we seek another way to use the semantic information, that is semantic-aware feature representation learning framework.Based on this, we propose SRMatcher, a new detector-free feature matching method, which encourages the network to learn integrated semantic feature representation.Specifically, to capture precise and rich semantics, we leverage the capabilities of recently popularized vision foundation models (VFMs) trained on extensive datasets. Then, a cross-images Semantic-aware Fusion Block (SFB) is proposed to integrate its fine-grained semantic features into the feature representation space. In this way, by reducing errors stemming from semantic inconsistencies in matching pairs, our proposed SRMatcher is able to deliver more accurate and realistic outcomes. Extensive experiments show that SRMatcher surpasses solid baselines and attains SOTA results on multiple real-world datasets. Compared to the previous SOTA approach GeoFormer, SRMatcher increases the area under the cumulative curve (AUC) by about 11% on HPatches. Additionally, the SRMatcher could serve as a plug-and-play framework for other matching methods like LoFTR, yielding substantial precision improvement.