Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization

TL;DR

This paper introduces a novel deep attention-based framework that combines appearance and semantic cues to improve 2D image-based localization, achieving significant accuracy gains over existing methods across challenging datasets.

Contribution

It proposes an end-to-end multimodal attention model that enhances localization robustness by focusing on reliable scene regions using semantic and appearance information.

Findings

19% average improvement over state-of-the-art methods

Semantic information adds 8-15% accuracy gain

Attention module contributes 4% accuracy improvement

Abstract

We present an approach that combines appearance and semantic information for 2D image-based localization (2D-VL) across large perceptual changes and time lags. Compared to appearance features, the semantic layout of a scene is generally more invariant to appearance variations. We use this intuition and propose a novel end-to-end deep attention-based framework that utilizes multimodal cues to generate robust embeddings for 2D-VL. The proposed attention module predicts a shared channel attention and modality-specific spatial attentions to guide the embeddings to focus on more reliable image regions. We evaluate our model against state-of-the-art (SOTA) methods on three challenging localization datasets. We report an average (absolute) improvement of $19\%$ over current SOTA for 2D-VL. Furthermore, we present an extensive study demonstrating the contribution of each component of our model, showing $8$--$15\%$ and $4\%$ improvement from adding semantic information and our proposed attention module. We finally show the predicted attention maps to offer useful insights into our model.