AFT-VO: Asynchronous Fusion Transformers for Multi-View Visual Odometry Estimation

TL;DR

AFT-VO introduces a transformer-based sensor fusion method for visual odometry that effectively handles asynchronous multi-view camera data, improving robustness and accuracy in challenging conditions.

Contribution

The paper presents a novel transformer architecture, AFT-VO, that fuses asynchronous multi-view camera data for visual odometry, addressing synchronization issues in sensor fusion.

Findings

Outperforms state-of-the-art methods on nuScenes and KITTI datasets.

Provides robust and accurate trajectories in challenging weather and lighting.

Effectively handles asynchronous sensor measurements.

Abstract

Motion estimation approaches typically employ sensor fusion techniques, such as the Kalman Filter, to handle individual sensor failures. More recently, deep learning-based fusion approaches have been proposed, increasing the performance and requiring less model-specific implementations. However, current deep fusion approaches often assume that sensors are synchronised, which is not always practical, especially for low-cost hardware. To address this limitation, in this work, we propose AFT-VO, a novel transformer-based sensor fusion architecture to estimate VO from multiple sensors. Our framework combines predictions from asynchronous multi-view cameras and accounts for the time discrepancies of measurements coming from different sources. Our approach first employs a Mixture Density Network (MDN) to estimate the probability distributions of the 6-DoF poses for every camera in the system. Then a novel transformer-based fusion module, AFT-VO, is introduced, which combines these asynchronous pose estimations, along with their confidences. More specifically, we introduce Discretiser and Source Encoding techniques which enable the fusion of multi-source asynchronous signals. We evaluate our approach on the popular nuScenes and KITTI datasets. Our experiments demonstrate that multi-view fusion for VO estimation provides robust and accurate trajectories, outperforming the state of the art in both challenging weather and lighting conditions.