RDD: Robust Feature Detector and Descriptor using Deformable Transformer

TL;DR

The paper introduces RDD, a deformable transformer-based feature detector and descriptor that captures global context and geometric invariance, improving robustness in challenging scenarios like large viewpoint changes.

Contribution

It proposes a novel deformable transformer-based method for robust feature detection and description, and introduces new datasets and benchmarks for evaluation.

Findings

Outperforms state-of-the-art methods in sparse matching tasks

Capable of semi-dense matching under challenging conditions

Effective in large viewpoint and scale variation scenarios

Abstract

As a core step in structure-from-motion and SLAM, robust feature detection and description under challenging scenarios such as significant viewpoint changes remain unresolved despite their ubiquity. While recent works have identified the importance of local features in modeling geometric transformations, these methods fail to learn the visual cues present in long-range relationships. We present Robust Deformable Detector (RDD), a novel and robust keypoint detector/descriptor leveraging the deformable transformer, which captures global context and geometric invariance through deformable self-attention mechanisms. Specifically, we observed that deformable attention focuses on key locations, effectively reducing the search space complexity and modeling the geometric invariance. Furthermore, we collected an Air-to-Ground dataset for training in addition to the standard MegaDepth dataset. Our proposed method outperforms all state-of-the-art keypoint detection/description methods in sparse matching tasks and is also capable of semi-dense matching. To ensure comprehensive evaluation, we introduce two challenging benchmarks: one emphasizing large viewpoint and scale variations, and the other being an Air-to-Ground benchmark -- an evaluation setting that has recently gaining popularity for 3D reconstruction across different altitudes.