LiftFeat: 3D Geometry-Aware Local Feature Matching

TL;DR

LiftFeat introduces a lightweight 3D geometry-aware network that enhances local feature matching robustness in challenging conditions by integrating surface normal information with 2D descriptors.

Contribution

The paper presents a novel method combining 3D geometric features with 2D descriptors using a surface normal supervision for improved feature robustness.

Findings

Outperforms lightweight state-of-the-art methods in pose and localization tasks.

Enhances feature discriminability in extreme lighting and texture conditions.

Demonstrates effectiveness across multiple visual estimation tasks.

Abstract

Robust and efficient local feature matching plays a crucial role in applications such as SLAM and visual localization for robotics. Despite great progress, it is still very challenging to extract robust and discriminative visual features in scenarios with drastic lighting changes, low texture areas, or repetitive patterns. In this paper, we propose a new lightweight network called \textit{LiftFeat}, which lifts the robustness of raw descriptor by aggregating 3D geometric feature. Specifically, we first adopt a pre-trained monocular depth estimation model to generate pseudo surface normal label, supervising the extraction of 3D geometric feature in terms of predicted surface normal. We then design a 3D geometry-aware feature lifting module to fuse surface normal feature with raw 2D descriptor feature. Integrating such 3D geometric feature enhances the discriminative ability of 2D feature description in extreme conditions. Extensive experimental results on relative pose estimation, homography estimation, and visual localization tasks, demonstrate that our LiftFeat outperforms some lightweight state-of-the-art methods. Code will be released at : https://github.com/lyp-deeplearning/LiftFeat.