A Generative Map for Image-based Camera Localization

TL;DR

This paper introduces a Generative Map framework that creates human-readable neural network maps for image-based camera localization, combining generative models with Kalman filtering to improve interpretability and incorporate sensor data.

Contribution

It presents a novel generative map approach that is interpretable, integrates sensor information, and is trained from scratch, unlike existing regression-based models.

Findings

Generative Map predicts images closely resembling true scenes.

Achieves localization performance comparable to current regression models.

Trained entirely from scratch without large pretrained networks.

Abstract

In image-based camera localization systems, information about the environment is usually stored in some representation, which can be referred to as a map. Conventionally, most maps are built upon hand-crafted features. Recently, neural networks have attracted attention as a data-driven map representation, and have shown promising results in visual localization. However, these neural network maps are generally hard to interpret by human. A readable map is not only accessible to humans, but also provides a way to be verified when the ground truth pose is unavailable. To tackle this problem, we propose Generative Map, a new framework for learning human-readable neural network maps, by combining a generative model with the Kalman filter, which also allows it to incorporate additional sensor information such as stereo visual odometry. For evaluation, we use real world images from the 7-Scenes and Oxford RobotCar datasets. We demonstrate that our Generative Map can be queried with a pose of interest from the test sequence to predict an image, which closely resembles the true scene. For localization, we show that Generative Map achieves comparable performance with current regression models. Moreover, our framework is trained completely from scratch, unlike regression models which rely on large ImageNet pretrained networks.