SceneGlue: Scene-Aware Transformer for Feature Matching without Scene-Level Annotation

TL;DR

SceneGlue introduces a scene-aware transformer framework for feature matching that leverages implicit and explicit scene information, trained without scene-level annotations, to improve accuracy and robustness in cross-view correspondence tasks.

Contribution

It proposes a novel hybridizable matching paradigm with a Visibility Transformer, enabling scene awareness without scene-level groundtruth annotations.

Findings

Outperforms traditional methods in homography and pose estimation.

Enhances robustness and interpretability in feature matching.

Source code is publicly available at the provided GitHub link.

Abstract

Local feature matching plays a critical role in understanding the correspondence between cross-view images. However, traditional methods are constrained by the inherent local nature of feature descriptors, limiting their ability to capture non-local scene information that is essential for accurate cross-view correspondence. In this paper, we introduce SceneGlue, a scene-aware feature matching framework designed to overcome these limitations. SceneGlue leverages a hybridizable matching paradigm that integrates implicit parallel attention and explicit cross-view visibility estimation. The parallel attention mechanism simultaneously exchanges information among local descriptors within and across images, enhancing the scene's global context. To further enrich the scene awareness, we propose the Visibility Transformer, which explicitly categorizes features into visible and invisible regions, providing an understanding of cross-view scene visibility. By combining explicit and implicit scene-level awareness, SceneGlue effectively compensates for the local descriptor constraints. Notably, SceneGlue is trained using only local feature matches, without requiring scene-level groundtruth annotations. This scene-aware approach not only improves accuracy and robustness but also enhances interpretability compared to traditional methods. Extensive experiments on applications such as homography estimation, pose estimation, image matching, and visual localization validate SceneGlue's superior performance. The source code is available at https://github.com/songlin-du/SceneGlue.