MobiFuse: A High-Precision On-device Depth Perception System with Multi-Data Fusion

TL;DR

MobiFuse is a novel on-device depth perception system that fuses dual RGB and ToF data with environmental error modeling to produce highly accurate depth maps, validated on a new dataset and effective for 3D tasks.

Contribution

We introduce MobiFuse, a multi-data fusion approach combining ToF, stereo, and error modeling for high-precision depth estimation on mobile devices.

Findings

Reduces depth errors by up to 77.7%

Generalizes well across diverse datasets

Improves performance in 3D reconstruction and segmentation

Abstract

We present MobiFuse, a high-precision depth perception system on mobile devices that combines dual RGB and Time-of-Flight (ToF) cameras. To achieve this, we leverage physical principles from various environmental factors to propose the Depth Error Indication (DEI) modality, characterizing the depth error of ToF and stereo-matching. Furthermore, we employ a progressive fusion strategy, merging geometric features from ToF and stereo depth maps with depth error features from the DEI modality to create precise depth maps. Additionally, we create a new ToF-Stereo depth dataset, RealToF, to train and validate our model. Our experiments demonstrate that MobiFuse excels over baselines by significantly reducing depth measurement errors by up to 77.7%. It also showcases strong generalization across diverse datasets and proves effectiveness in two downstream tasks: 3D reconstruction and 3D segmentation. The demo video of MobiFuse in real-life scenarios is available at the de-identified YouTube link(https://youtu.be/jy-Sp7T1LVs).