REST: Real-to-Synthetic Transform for Illumination Invariant Camera Localization

TL;DR

This paper introduces REST, a neural transform that converts real image features into synthetic-like features, significantly improving illumination-invariant camera localization accuracy and robustness by bridging the gap between real and synthetic images.

Contribution

REST is a novel autoencoder-like network that reduces the real-synthetic feature gap, enhancing camera localization under varying lighting conditions.

Findings

REST improved feature matching accuracy by ~30% under variable lighting.

The system outperforms existing CNN-based localization methods trained on synthetic data.

REST facilitates lighting-robust localization and simplifies data collection.

Abstract

Accurate camera localization is an essential part of tracking systems. However, localization results are greatly affected by illumination. Including data collected under various lighting conditions can improve the robustness of the localization algorithm to lighting variation. However, this is very tedious and time consuming. By using synthesized images it is possible to easily accumulate a large variety of views under varying illumination and weather conditions. Despite continuously improving processing power and rendering algorithms, synthesized images do not perfectly match real images of the same scene, i.e. there exists a gap between real and synthesized images that also affects the accuracy of camera localization. To reduce the impact of this gap, we introduce "REal-to-Synthetic Transform (REST)." REST is an autoencoder-like network that converts real features to their synthetic counterpart. The converted features can then be matched against the accumulated database for robust camera localization. In our experiments REST improved feature matching accuracy under variable lighting conditions by approximately 30%. Moreover, our system outperforms state of the art CNN-based camera localization methods trained with synthetic images. We believe our method could be used to initialize local tracking and to simplify data accumulation for lighting robust localization.