Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor

TL;DR

This paper introduces a novel dense SLAM system combining monocular camera data with lightweight ToF sensors, utilizing a multi-modal implicit scene representation and a coarse-to-fine optimization to improve robustness and accuracy despite noisy measurements.

Contribution

It presents the first dense SLAM approach integrating lightweight ToF sensors with monocular cameras using a multi-modal implicit scene model and temporal information exploitation.

Findings

Achieves competitive camera tracking accuracy.

Demonstrates effective dense scene reconstruction.

Robust to noisy ToF sensor signals.

Abstract

Light-weight time-of-flight (ToF) depth sensors are compact and cost-efficient, and thus widely used on mobile devices for tasks such as autofocus and obstacle detection. However, due to the sparse and noisy depth measurements, these sensors have rarely been considered for dense geometry reconstruction. In this work, we present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. Specifically, we propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor which drives the optimization by comparing with the raw sensor inputs. Moreover, in order to guarantee successful pose tracking and reconstruction, we exploit a predicted depth as an intermediate supervision and develop a coarse-to-fine optimization strategy for efficient learning of the implicit representation. At last, the temporal information is explicitly exploited to deal with the noisy signals from light-weight ToF sensors to improve the accuracy and robustness of the system. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results both on camera tracking and dense scene reconstruction. Project page: \url{https://zju3dv.github.io/tof_slam/}.