3DG-STFM: 3D Geometric Guided Student-Teacher Feature Matching

TL;DR

This paper introduces 3DG-STFM, a novel student-teacher transformer-based model that leverages 3D dense correspondence supervision to improve dense visual matching accuracy in various challenging scenarios.

Contribution

It presents the first student-teacher learning framework for local feature matching, utilizing depth-guided supervision to enhance 2D feature matching performance.

Findings

Outperforms state-of-the-art methods in camera pose estimation

Effective in indoor and outdoor homography estimation

Improves dense correspondence accuracy under challenging conditions

Abstract

We tackle the essential task of finding dense visual correspondences between a pair of images. This is a challenging problem due to various factors such as poor texture, repetitive patterns, illumination variation, and motion blur in practical scenarios. In contrast to methods that use dense correspondence ground-truths as direct supervision for local feature matching training, we train 3DG-STFM: a multi-modal matching model (Teacher) to enforce the depth consistency under 3D dense correspondence supervision and transfer the knowledge to 2D unimodal matching model (Student). Both teacher and student models consist of two transformer-based matching modules that obtain dense correspondences in a coarse-to-fine manner. The teacher model guides the student model to learn RGB-induced depth information for the matching purpose on both coarse and fine branches. We also evaluate 3DG-STFM on a model compression task. To the best of our knowledge, 3DG-STFM is the first student-teacher learning method for the local feature matching task. The experiments show that our method outperforms state-of-the-art methods on indoor and outdoor camera pose estimations, and homography estimation problems. Code is available at: https://github.com/Ryan-prime/3DG-STFM.