XVO: Generalized Visual Odometry via Cross-Modal Self-Training

TL;DR

XVO introduces a semi-supervised, multi-modal approach to monocular visual odometry that learns from diverse, real-world dashcam videos without relying on camera calibration, achieving robust, generalized pose estimation.

Contribution

The paper presents a novel semi-supervised learning framework with multi-modal supervision for generalized monocular visual odometry, enabling off-the-shelf performance across datasets.

Findings

Achieves state-of-the-art results on KITTI without multi-frame optimization.

Effectively transfers knowledge across diverse datasets without fine-tuning.

Audio auxiliary tasks significantly improve learning in dynamic, out-of-domain videos.

Abstract

We propose XVO, a semi-supervised learning method for training generalized monocular Visual Odometry (VO) models with robust off-the-self operation across diverse datasets and settings. In contrast to standard monocular VO approaches which often study a known calibration within a single dataset, XVO efficiently learns to recover relative pose with real-world scale from visual scene semantics, i.e., without relying on any known camera parameters. We optimize the motion estimation model via self-training from large amounts of unconstrained and heterogeneous dash camera videos available on YouTube. Our key contribution is twofold. First, we empirically demonstrate the benefits of semi-supervised training for learning a general-purpose direct VO regression network. Second, we demonstrate multi-modal supervision, including segmentation, flow, depth, and audio auxiliary prediction tasks, to facilitate generalized representations for the VO task. Specifically, we find audio prediction task to significantly enhance the semi-supervised learning process while alleviating noisy pseudo-labels, particularly in highly dynamic and out-of-domain video data. Our proposed teacher network achieves state-of-the-art performance on the commonly used KITTI benchmark despite no multi-frame optimization or knowledge of camera parameters. Combined with the proposed semi-supervised step, XVO demonstrates off-the-shelf knowledge transfer across diverse conditions on KITTI, nuScenes, and Argoverse without fine-tuning.