Blur aware metric depth estimation with multi-focus plenoptic cameras

TL;DR

This paper introduces BLADE, a novel depth estimation method for multi-focus plenoptic cameras that leverages defocus blur cues to improve accuracy and provides a calibration approach for metric depth estimation.

Contribution

The paper presents a new blur-aware depth estimation algorithm that explicitly models defocus blur and calibrates depth scaling for accurate metric depth from multi-focus plenoptic images.

Findings

Incorporating defocus cues enhances depth estimation accuracy.

The proposed calibration method achieves precise metric depth.

Validated on real-world scenes with lidar ground truth.

Abstract

While a traditional camera only captures one point of view of a scene, a plenoptic or light-field camera, is able to capture spatial and angular information in a single snapshot, enabling depth estimation from a single acquisition. In this paper, we present a new metric depth estimation algorithm using only raw images from a multi-focus plenoptic camera. The proposed approach is especially suited for the multi-focus configuration where several micro-lenses with different focal lengths are used. The main goal of our blur aware depth estimation (BLADE) approach is to improve disparity estimation for defocus stereo images by integrating both correspondence and defocus cues. We thus leverage blur information where it was previously considered a drawback. We explicitly derive an inverse projection model including the defocus blur providing depth estimates up to a scale factor. A method to calibrate the inverse model is then proposed. We thus take into account depth scaling to achieve precise and accurate metric depth estimates. Our results show that introducing defocus cues improves the depth estimation. We demonstrate the effectiveness of our framework and depth scaling calibration on relative depth estimation setups and on real-world 3D complex scenes with ground truth acquired with a 3D lidar scanner.