# Learning Matchable Image Transformations for Long-term Metric Visual Localization

## TL;DR

This paper introduces a learned RGB-to-grayscale transformation inspired by color constancy theory to enhance long-term visual localization robustness against appearance changes, reducing the need for extensive experience data.

## Contribution

It proposes a novel nonlinear RGB-to-grayscale mapping learned via neural networks to improve feature matching under varying conditions, enabling scalable long-term localization.

## Key findings

- Significant performance improvements over day-night cycles.
- Achieves continuous localization over 30 hours with a single experience.
- Reduces data requirements for experience-based localization.

## Abstract

Long-term metric self-localization is an essential capability of autonomous mobile robots, but remains challenging for vision-based systems due to appearance changes caused by lighting, weather, or seasonal variations. While experience-based mapping has proven to be an effective technique for bridging the `appearance gap,' the number of experiences required for reliable metric localization over days or months can be very large, and methods for reducing the necessary number of experiences are needed for this approach to scale. Taking inspiration from color constancy theory, we learn a nonlinear RGB-to-grayscale mapping that explicitly maximizes the number of inlier feature matches for images captured under different lighting and weather conditions, and use it as a pre-processing step in a conventional single-experience localization pipeline to improve its robustness to appearance change. We train this mapping by approximating the target non-differentiable localization pipeline with a deep neural network, and find that incorporating a learned low-dimensional context feature can further improve cross-appearance feature matching. Using synthetic and real-world datasets, we demonstrate substantial improvements in localization performance across day-night cycles, enabling continuous metric localization over a 30-hour period using a single mapping experience, and allowing experience-based localization to scale to long deployments with dramatically reduced data requirements.