XRefine: Attention-Guided Keypoint Match Refinement

TL;DR

XRefine is a detector-agnostic, attention-based method for sub-pixel keypoint refinement that improves 3D vision accuracy across various datasets without needing detector retraining.

Contribution

We propose XRefine, a novel cross-attention architecture that refines keypoints using image patches, enabling detector-agnostic and multi-view compatible keypoint refinement.

Findings

Consistently improves geometric estimation accuracy on multiple datasets.

Outperforms existing refinement methods in accuracy and efficiency.

Generalizes across different keypoint detectors without retraining.

Abstract

Sparse keypoint matching is crucial for 3D vision tasks, yet current keypoint detectors often produce spatially inaccurate matches. Existing refinement methods mitigate this issue through alignment of matched keypoint locations, but they are typically detector-specific, requiring retraining for each keypoint detector. We introduce XRefine, a novel, detector-agnostic approach for sub-pixel keypoint refinement that operates solely on image patches centered at matched keypoints. Our cross-attention-based architecture learns to predict refined keypoint coordinates without relying on internal detector representations, enabling generalization across detectors. Furthermore, XRefine can be extended to handle multi-view feature tracks. Experiments on MegaDepth, KITTI, and ScanNet demonstrate that the approach consistently improves geometric estimation accuracy, achieving superior performance compared to existing refinement methods while maintaining runtime efficiency. Our code and trained models can be found at https://github.com/boschresearch/xrefine.