SPIDER: Spatial Image CorresponDence Estimator for Robust Calibration

TL;DR

SPIDER is a universal image-matching framework that combines 2D and 3D correspondence estimation to improve robustness and accuracy in unconstrained scenarios with large viewpoint changes.

Contribution

The paper introduces SPIDER, a novel framework integrating 2D and 3D matching with a shared backbone and specialized heads, and provides a new benchmark for challenging image-matching scenarios.

Findings

SPIDER outperforms state-of-the-art methods significantly.

It effectively handles large viewpoint variations.

The benchmark reveals the strengths of SPIDER in unconstrained environments.

Abstract

Reliable image correspondences form the foundation of vision-based spatial perception, enabling recovery of 3D structure and camera poses. However, unconstrained feature matching across domains such as aerial, indoor, and outdoor scenes remains challenging due to large variations in appearance, scale and viewpoint. Feature matching has been conventionally formulated as a 2D-to-2D problem; however, recent 3D foundation models provides spatial feature matching properties based on two-view geometry. While powerful, we observe that these spatially coherent matches often concentrate on dominant planar regions, e.g., walls or ground surfaces, while being less sensitive to fine-grained geometric details, particularly under large viewpoint changes. To better understand these trade-offs, we first perform linear probe experiments to evaluate the performance of various vision foundation models for image matching. Building on these insights, we introduce SPIDER, a universal feature matching framework that integrates a shared feature extraction backbone with two specialized network heads for estimating both 2D-based and 3D-based correspondences from coarse to fine. Finally, we introduce an image-matching evaluation benchmark that focuses on unconstrained scenarios with large baselines. SPIDER significantly outperforms SoTA methods, demonstrating its strong ability as a universal image-matching method.