Structure-From-Motion and RGBD Depth Fusion

TL;DR

This paper presents a method to enhance RGBD sensor depth data by fusing it with Structure-from-Motion estimates, overcoming limitations in challenging environments for improved robotic and computer vision applications.

Contribution

The paper introduces a novel fusion technique combining RGBD sensor data with SfM depth estimates to improve depth sensing in difficult conditions.

Findings

Enhanced depth accuracy in distant, dark, and brightly lit scenes.

Robust depth estimation across various challenging environments.

Potential improvements in robotic localization and object tracking.

Abstract

This article describes a technique to augment a typical RGBD sensor by integrating depth estimates obtained via Structure-from-Motion (SfM) with sensor depth measurements. Limitations in the RGBD depth sensing technology prevent capturing depth measurements in four important contexts: (1) distant surfaces (>5m), (2) dark surfaces, (3) brightly lit indoor scenes and (4) sunlit outdoor scenes. SfM technology computes depth via multi-view reconstruction from the RGB image sequence alone. As such, SfM depth estimates do not suffer the same limitations and may be computed in all four of the previously listed circumstances. This work describes a novel fusion of RGBD depth data and SfM-estimated depths to generate an improved depth stream that may be processed by one of many important downstream applications such as robotic localization and mapping, as well as object recognition and tracking.