D2S: Representing sparse descriptors and 3D coordinates for camera relocalization

TL;DR

D2S introduces a simple, cost-effective learning-based method for camera relocalization that uses a lightweight network to represent descriptors and scene coordinates, outperforming previous regression methods in various environments.

Contribution

The paper presents D2S, a novel approach that leverages a simple network with graph attention for efficient, scene-specific localization from a single RGB image, with improved generalization capabilities.

Findings

Outperforms previous regression-based methods in indoor and outdoor environments.

Effectively generalizes across day-night transitions and domain shifts.

Uses a lightweight model with selective attention for robust descriptor representation.

Abstract

State-of-the-art visual localization methods mostly rely on complex procedures to match local descriptors and 3D point clouds. However, these procedures can incur significant costs in terms of inference, storage, and updates over time. In this study, we propose a direct learning-based approach that utilizes a simple network named D2S to represent complex local descriptors and their scene coordinates. Our method is characterized by its simplicity and cost-effectiveness. It solely leverages a single RGB image for localization during the testing phase and only requires a lightweight model to encode a complex sparse scene. The proposed D2S employs a combination of a simple loss function and graph attention to selectively focus on robust descriptors while disregarding areas such as clouds, trees, and several dynamic objects. This selective attention enables D2S to effectively perform a binary-semantic classification for sparse descriptors. Additionally, we propose a simple outdoor dataset to evaluate the capabilities of visual localization methods in scene-specific generalization and self-updating from unlabeled observations. Our approach outperforms the previous regression-based methods in both indoor and outdoor environments. It demonstrates the ability to generalize beyond training data, including scenarios involving transitions from day to night and adapting to domain shifts. The source code, trained models, dataset, and demo videos are available at the following link: https://thpjp.github.io/d2s.