Single-Shot Metric Depth from Focused Plenoptic Cameras

TL;DR

This paper introduces a novel method for dense metric depth estimation using focused plenoptic cameras, combining machine learning and depth scaling techniques, validated on a new real-world light field dataset.

Contribution

It presents a new pipeline for metric depth from a single focused plenoptic camera shot, including dataset creation, addressing a gap in current light field depth estimation research.

Findings

Accurate dense metric depth predictions achieved.

New Light Field & Stereo Image Dataset (LFS) curated.

Pipeline outperforms existing methods in accuracy.

Abstract

Metric depth estimation from visual sensors is crucial for robots to perceive, navigate, and interact with their environment. Traditional range imaging setups, such as stereo or structured light cameras, face hassles including calibration, occlusions, and hardware demands, with accuracy limited by the baseline between cameras. Single- and multi-view monocular depth offers a more compact alternative, but is constrained by the unobservability of the metric scale. Light field imaging provides a promising solution for estimating metric depth by using a unique lens configuration through a single device. However, its application to single-view dense metric depth is under-addressed mainly due to the technology's high cost, the lack of public benchmarks, and proprietary geometrical models and software. Our work explores the potential of focused plenoptic cameras for dense metric depth. We propose a novel pipeline that predicts metric depth from a single plenoptic camera shot by first generating a sparse metric point cloud using machine learning, which is then used to scale and align a dense relative depth map regressed by a foundation depth model, resulting in dense metric depth. To validate it, we curated the Light Field & Stereo Image Dataset (LFS) of real-world light field images with stereo depth labels, filling a current gap in existing resources. Experimental results show that our pipeline produces accurate metric depth predictions, laying a solid groundwork for future research in this field.