What to Learn: Features, Image Transformations, or Both?

TL;DR

This paper explores combining neural style transfer with feature learning to enhance long-term visual localization under environmental appearance changes, demonstrating significant performance improvements.

Contribution

It introduces a novel approach that integrates image transformation and feature learning networks for better long-term localization accuracy.

Findings

Combining style transfer with feature learning improves localization performance.

Transforming night images to day-like conditions aids in feature matching.

The proposed training strategy enhances robustness to appearance changes.

Abstract

Long-term visual localization is an essential problem in robotics and computer vision, but remains challenging due to the environmental appearance changes caused by lighting and seasons. While many existing works have attempted to solve it by directly learning invariant sparse keypoints and descriptors to match scenes, these approaches still struggle with adverse appearance changes. Recent developments in image transformations such as neural style transfer have emerged as an alternative to address such appearance gaps. In this work, we propose to combine an image transformation network and a feature-learning network to improve long-term localization performance. Given night-to-day image pairs, the image transformation network transforms the night images into day-like conditions prior to feature matching; the feature network learns to detect keypoint locations with their associated descriptor values, which can be passed to a classical pose estimator to compute the relative poses. We conducted various experiments to examine the effectiveness of combining style transfer and feature learning and its training strategy, showing that such a combination greatly improves long-term localization performance.