Adversarial Exploitation of Data Diversity Improves Visual Localization

TL;DR

This paper introduces a novel data augmentation technique leveraging appearance diversity and adversarial training to significantly improve the robustness and accuracy of visual localization methods in diverse environmental conditions.

Contribution

It presents a new approach that synthesizes appearance-diverse training data using 3D Gaussian Splats and employs adversarial training to enhance generalization in visual localization.

Findings

Reduces translation and rotation errors by over 40% on benchmark datasets.

Demonstrates robustness in dynamic and varying weather conditions.

Outperforms state-of-the-art methods in indoor and outdoor localization tasks.

Abstract

Visual localization, which estimates a camera's pose within a known scene, is a fundamental capability for autonomous systems. While absolute pose regression (APR) methods have shown promise for efficient inference, they often struggle with generalization. Recent approaches attempt to address this through data augmentation with varied viewpoints, yet they overlook a critical factor: appearance diversity. In this work, we identify appearance variation as the key to robust localization. Specifically, we first lift real 2D images into 3D Gaussian Splats with varying appearance and deblurring ability, enabling the synthesis of diverse training data that varies not just in poses but also in environmental conditions such as lighting and weather. To fully unleash the potential of the appearance-diverse data, we build a two-branch joint training pipeline with an adversarial discriminator to bridge the syn-to-real gap. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods, reducing translation and rotation errors by 50\% and 41\% on indoor datasets, and 38\% and 44\% on outdoor datasets. Most notably, our method shows remarkable robustness in dynamic driving scenarios under varying weather conditions and in day-to-night scenarios, where previous APR methods fail. Project Page: https://ai4ce.github.io/RAP/