Towards Multimodal Depth Estimation from Light Fields

TL;DR

This paper introduces a novel deep learning approach for multimodal depth estimation from light fields, addressing limitations of single-depth models by predicting depth distributions and providing a new dataset for training and validation.

Contribution

It proposes multiple deep-learning methods for multimodal depth estimation and introduces the first dataset containing all contributing object depths per pixel.

Findings

Deep models predict richer depth distributions.

The dataset enables supervised training and KL divergence validation.

Results show improved handling of transparent and reflective objects.

Abstract

Light field applications, especially light field rendering and depth estimation, developed rapidly in recent years. While state-of-the-art light field rendering methods handle semi-transparent and reflective objects well, depth estimation methods either ignore these cases altogether or only deliver a weak performance. We argue that this is due current methods only considering a single "true" depth, even when multiple objects at different depths contributed to the color of a single pixel. Based on the simple idea of outputting a posterior depth distribution instead of only a single estimate, we develop and explore several different deep-learning-based approaches to the problem. Additionally, we contribute the first "multimodal light field depth dataset" that contains the depths of all objects which contribute to the color of a pixel. This allows us to supervise the multimodal depth prediction and also validate all methods by measuring the KL divergence of the predicted posteriors. With our thorough analysis and novel dataset, we aim to start a new line of depth estimation research that overcomes some of the long-standing limitations of this field.