FMRT: Learning Accurate Feature Matching with Reconciliatory Transformer

TL;DR

FMRT introduces a novel Transformer-based approach that adaptively reconciles multi-scale features and enhances positional encoding, significantly improving local feature matching accuracy in various computer vision tasks.

Contribution

The paper presents FMRT, a new detector-free Transformer method with a Reconciliatory Transformer that adaptively integrates multi-scale features and reliable positional encoding.

Findings

FMRT outperforms existing methods on multiple benchmarks.

It achieves higher accuracy in pose estimation and visual localization.

The approach demonstrates robustness across diverse computer vision tasks.

Abstract

Local Feature Matching, an essential component of several computer vision tasks (e.g., structure from motion and visual localization), has been effectively settled by Transformer-based methods. However, these methods only integrate long-range context information among keypoints with a fixed receptive field, which constrains the network from reconciling the importance of features with different receptive fields to realize complete image perception, hence limiting the matching accuracy. In addition, these methods utilize a conventional handcrafted encoding approach to integrate the positional information of keypoints into the visual descriptors, which limits the capability of the network to extract reliable positional encoding message. In this study, we propose Feature Matching with Reconciliatory Transformer (FMRT), a novel Transformer-based detector-free method that reconciles different features with multiple receptive fields adaptively and utilizes parallel networks to realize reliable positional encoding. Specifically, FMRT proposes a dedicated Reconciliatory Transformer (RecFormer) that consists of a Global Perception Attention Layer (GPAL) to extract visual descriptors with different receptive fields and integrate global context information under various scales, Perception Weight Layer (PWL) to measure the importance of various receptive fields adaptively, and Local Perception Feed-forward Network (LPFFN) to extract deep aggregated multi-scale local feature representation. Extensive experiments demonstrate that FMRT yields extraordinary performance on multiple benchmarks, including pose estimation, visual localization, homography estimation, and image matching.