Robust Visual Localization via Semantic-Guided Multi-Scale Transformer

TL;DR

This paper introduces a novel semantic-guided multi-scale Transformer framework that significantly improves visual localization accuracy in dynamic environments by combining hierarchical feature learning with semantic scene understanding.

Contribution

It presents a new approach that integrates multi-scale Transformer architecture with semantic supervision to enhance robustness in visual localization under challenging conditions.

Findings

Outperforms existing pose regression methods in dynamic scenarios

Effective in handling lighting changes, occlusions, and moving objects

Demonstrates robustness on TartanAir dataset

Abstract

Visual localization remains challenging in dynamic environments where fluctuating lighting, adverse weather, and moving objects disrupt appearance cues. Despite advances in feature representation, current absolute pose regression methods struggle to maintain consistency under varying conditions. To address this challenge, we propose a framework that synergistically combines multi-scale feature learning with semantic scene understanding. Our approach employs a hierarchical Transformer with cross-scale attention to fuse geometric details and contextual cues, preserving spatial precision while adapting to environmental changes. We improve the performance of this architecture with semantic supervision via neural scene representation during training, guiding the network to learn view-invariant features that encode persistent structural information while suppressing complex environmental interference. Experiments on TartanAir demonstrate that our approach outperforms existing pose regression methods in challenging scenarios with dynamic objects, illumination changes, and occlusions. Our findings show that integrating multi-scale processing with semantic guidance offers a promising strategy for robust visual localization in real-world dynamic environments.