NeRF-VINS: A Real-time Neural Radiance Field Map-based Visual-Inertial Navigation System

TL;DR

This paper introduces NeRF-VINS, a real-time visual-inertial navigation system that leverages neural radiance fields to synthesize novel views, overcoming traditional map limitations for improved localization in robotics.

Contribution

The paper presents a novel NeRF-aided VINS that fuses IMU, images, and synthetic views in real-time, enhancing localization accuracy and robustness over traditional methods.

Findings

Achieves real-time localization at over 10 Hz on embedded hardware.

Outperforms state-of-the-art methods using prior maps.

Effectively fuses synthetic and real data for improved motion tracking.

Abstract

Achieving efficient and consistent localization a prior map remains challenging in robotics. Conventional keyframe-based approaches often suffers from sub-optimal viewpoints due to limited field of view (FOV) and/or constrained motion, thus degrading the localization performance. To address this issue, we design a real-time tightly-coupled Neural Radiance Fields (NeRF)-aided visual-inertial navigation system (VINS). In particular, by effectively leveraging the NeRF's potential to synthesize novel views, the proposed NeRF-VINS overcomes the limitations of traditional keyframe-based maps (with limited views) and optimally fuses IMU, monocular images, and synthetically rendered images within an efficient filter-based framework. This tightly-coupled fusion enables efficient 3D motion tracking with bounded errors. We extensively compare the proposed NeRF-VINS against the state-of-the-art methods that use prior map information and demonstrate its ability to perform real-time localization, at over 10 Hz, on a resource-constrained Jetson AGX Orin embedded platform.