MESA: Effective Matching Redundancy Reduction by Semantic Area Segmentation

TL;DR

MESA and DMESA are novel feature matching methods that leverage the Segment Anything Model to reduce redundancy by semantic area segmentation, improving accuracy and efficiency in point matching across diverse datasets.

Contribution

The paper introduces MESA and DMESA, novel semantic area-based feature matching methods that enhance matching accuracy and efficiency by integrating SAM and graph-based optimization.

Findings

DMESA achieves nearly five times faster matching than MESA.

Both methods improve accuracy across five datasets.

Methods show robustness to image resolution changes.

Abstract

We propose MESA and DMESA as novel feature matching methods, which utilize Segment Anything Model (SAM) to effectively mitigate matching redundancy. The key insight of our methods is to establish implicit-semantic area matching prior to point matching, based on advanced image understanding of SAM. Then, informative area matches with consistent internal semantic are able to undergo dense feature comparison, facilitating precise inside-area point matching. Specifically, MESA adopts a sparse matching framework and first obtains candidate areas from SAM results through a novel Area Graph (AG). Then, area matching among the candidates is formulated as graph energy minimization and solved by graphical models derived from AG. To address the efficiency issue of MESA, we further propose DMESA as its dense counterpart, applying a dense matching framework. After candidate areas are identified by AG, DMESA establishes area matches through generating dense matching distributions. The distributions are produced from off-the-shelf patch matching utilizing the Gaussian Mixture Model and refined via the Expectation Maximization. With less repetitive computation, DMESA showcases a speed improvement of nearly five times compared to MESA, while maintaining competitive accuracy. Our methods are extensively evaluated on five datasets encompassing indoor and outdoor scenes. The results illustrate consistent performance improvements from our methods for five distinct point matching baselines across all datasets. Furthermore, our methods exhibit promise generalization and improved robustness against image resolution variations. The code is publicly available at https://github.com/Easonyesheng/A2PM-MESA.