FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras

TL;DR

This paper introduces a novel calibration and fusion method combining ToF sensors and stabilized stereo cameras to improve high-resolution depth estimation on smartphones, using deep learning and dense matching.

Contribution

We develop an automatic calibration technique for stabilized RGB cameras and fuse ToF and stereo data through deep learning for enhanced depth accuracy.

Findings

Achieves higher depth accuracy than existing methods.

Effective calibration from a single snapshot.

Improved depth estimation in real-world scenarios.

Abstract

High-accuracy per-pixel depth is vital for computational photography, so smartphones now have multimodal camera systems with time-of-flight (ToF) depth sensors and multiple color cameras. However, producing accurate high-resolution depth is still challenging due to the low resolution and limited active illumination power of ToF sensors. Fusing RGB stereo and ToF information is a promising direction to overcome these issues, but a key problem remains: to provide high-quality 2D RGB images, the main color sensor's lens is optically stabilized, resulting in an unknown pose for the floating lens that breaks the geometric relationships between the multimodal image sensors. Leveraging ToF depth estimates and a wide-angle RGB camera, we design an automatic calibration technique based on dense 2D/3D matching that can estimate camera extrinsic, intrinsic, and distortion parameters of a stabilized main RGB sensor from a single snapshot. This lets us fuse stereo and ToF cues via a correlation volume. For fusion, we apply deep learning via a real-world training dataset with depth supervision estimated by a neural reconstruction method. For evaluation, we acquire a test dataset using a commercial high-power depth camera and show that our approach achieves higher accuracy than existing baselines.