SC-wLS: Towards Interpretable Feed-forward Camera Re-localization

TL;DR

SC-wLS introduces a differentiable, feed-forward camera re-localization method that combines scene coordinate accuracy with interpretability and enables self-supervised test-time adaptation, improving performance over previous methods.

Contribution

The paper presents SC-wLS, a novel differentiable, feed-forward approach that leverages all scene coordinate estimates for weighted least squares pose regression, enhancing accuracy and interpretability.

Findings

Significantly improved re-localization accuracy on 7Scenes and Cambridge datasets.

Demonstrates interpretability of learned weights in the model.

Enables self-supervised test-time adaptation for further performance gains.

Abstract

Visual re-localization aims to recover camera poses in a known environment, which is vital for applications like robotics or augmented reality. Feed-forward absolute camera pose regression methods directly output poses by a network, but suffer from low accuracy. Meanwhile, scene coordinate based methods are accurate, but need iterative RANSAC post-processing, which brings challenges to efficient end-to-end training and inference. In order to have the best of both worlds, we propose a feed-forward method termed SC-wLS that exploits all scene coordinate estimates for weighted least squares pose regression. This differentiable formulation exploits a weight network imposed on 2D-3D correspondences, and requires pose supervision only. Qualitative results demonstrate the interpretability of learned weights. Evaluations on 7Scenes and Cambridge datasets show significantly promoted performance when compared with former feed-forward counterparts. Moreover, our SC-wLS method enables a new capability: self-supervised test-time adaptation on the weight network. Codes and models are publicly available.