Self-Improving Visual Odometry

TL;DR

This paper introduces a self-supervised learning framework for visual odometry that improves over time by retraining on large-scale unlabeled monocular video data, outperforming traditional and deep learning methods.

Contribution

It presents a novel self-supervised approach enabling a VO frontend to learn and adapt over time without manual tuning or labeled data.

Findings

Outperforms traditional feature descriptors like SIFT, ORB, AKAZE.

Outperforms deep learning methods SuperPoint and LF-Net.

Automatically identifies less useful keypoints such as shadows and dynamic objects.

Abstract

We propose a self-supervised learning framework that uses unlabeled monocular video sequences to generate large-scale supervision for training a Visual Odometry (VO) frontend, a network which computes pointwise data associations across images. Our self-improving method enables a VO frontend to learn over time, unlike other VO and SLAM systems which require time-consuming hand-tuning or expensive data collection to adapt to new environments. Our proposed frontend operates on monocular images and consists of a single multi-task convolutional neural network which outputs 2D keypoints locations, keypoint descriptors, and a novel point stability score. We use the output of VO to create a self-supervised dataset of point correspondences to retrain the frontend. When trained using VO at scale on 2.5 million monocular images from ScanNet, the stability classifier automatically discovers a ranking for keypoints that are not likely to help in VO, such as t-junctions across depth discontinuities, features on shadows and highlights, and dynamic objects like people. The resulting frontend outperforms both traditional methods (SIFT, ORB, AKAZE) and deep learning methods (SuperPoint and LF-Net) in a 3D-to-2D pose estimation task on ScanNet.