Keeping an Eye on Things: Deep Learned Features for Long-Term Visual Localization

TL;DR

This paper introduces a neural network-based visual feature learning method for long-term autonomous robot localization, capable of handling lighting changes and darkness, validated through extensive outdoor experiments.

Contribution

We develop a training pipeline with a differentiable pose estimator to learn robust features for long-term visual localization in outdoor environments.

Findings

Successful path following across all lighting conditions

Features generalize to new areas not in training data

Validated over 35.5 km of autonomous navigation

Abstract

In this paper, we learn visual features that we use to first build a map and then localize a robot driving autonomously across a full day of lighting change, including in the dark. We train a neural network to predict sparse keypoints with associated descriptors and scores that can be used together with a classical pose estimator for localization. Our training pipeline includes a differentiable pose estimator such that training can be supervised with ground truth poses from data collected earlier, in our case from 2016 and 2017 gathered with multi-experience Visual Teach and Repeat (VT&R). We insert the learned features into the existing VT&R pipeline to perform closed-loop path following in unstructured outdoor environments. We show successful path following across all lighting conditions despite the robot's map being constructed using daylight conditions. Moreover, we explore generalizability of the features by driving the robot across all lighting conditions in new areas not present in the feature training dataset. In all, we validated our approach with 35.5 km of autonomous path following experiments in challenging conditions.