Continual Learning for Image-Based Camera Localization

TL;DR

This paper explores continual learning for visual localization, addressing catastrophic forgetting by proposing a buffer-based replay method and a coverage-based sampling strategy, improving localization accuracy across multiple datasets.

Contribution

It introduces a novel sampling method (Buff-CS) and demonstrates effective continual learning for camera localization with deep networks.

Findings

Buffer replay mitigates catastrophic forgetting.

Coverage score sampling improves localization accuracy.

Method outperforms standard buffering on multiple datasets.

Abstract

For several emerging technologies such as augmented reality, autonomous driving and robotics, visual localization is a critical component. Directly regressing camera pose/3D scene coordinates from the input image using deep neural networks has shown great potential. However, such methods assume a stationary data distribution with all scenes simultaneously available during training. In this paper, we approach the problem of visual localization in a continual learning setup -- whereby the model is trained on scenes in an incremental manner. Our results show that similar to the classification domain, non-stationary data induces catastrophic forgetting in deep networks for visual localization. To address this issue, a strong baseline based on storing and replaying images from a fixed buffer is proposed. Furthermore, we propose a new sampling method based on coverage score (Buff-CS) that adapts the existing sampling strategies in the buffering process to the problem of visual localization. Results demonstrate consistent improvements over standard buffering methods on two challenging datasets -- 7Scenes, 12Scenes, and also 19Scenes by combining the former scenes.