Dynamic Fusion Network For Light Field Depth Estimation

TL;DR

This paper introduces a dynamic multi-modal fusion network that combines RGB images and focal stacks to improve light field depth estimation, overcoming focus limitations and enabling practical use with consumer cameras.

Contribution

It proposes a novel adaptive fusion framework that effectively integrates RGB and focal stack data for enhanced depth estimation.

Findings

Achieved state-of-the-art performance on two datasets.

Demonstrated effectiveness on smartphone-generated focused images.

Overcame focus dependence limitations of previous methods.

Abstract

Focus based methods have shown promising results for the task of depth estimation. However, most existing focus based depth estimation approaches depend on maximal sharpness of the focal stack. Out of focus information in the focal stack poses challenges for this task. In this paper, we propose a dynamically multi modal learning strategy which incorporates RGB data and the focal stack in our framework. Our goal is to deeply excavate the spatial correlation in the focal stack by designing the spatial correlation perception module and dynamically fuse multi modal information between RGB data and the focal stack in a adaptive way by designing the multi modal dynamic fusion module. The success of our method is demonstrated by achieving the state of the art performance on two datasets. Furthermore, we test our network on a set of different focused images generated by a smart phone camera to prove that the proposed method not only broke the limitation of only using light field data, but also open a path toward practical applications of depth estimation on common consumer level cameras data.