Multi-task Learning for Monocular Depth and Defocus Estimations with Real Images

TL;DR

This paper introduces a multi-task learning network that jointly estimates depth and defocus from a single image, leveraging their physical connection and a new large real-image dataset to improve accuracy over existing methods.

Contribution

The work presents a novel multi-task network architecture and the first real-image dataset for joint depth and defocus estimation, enhancing performance through task mutual facilitation.

Findings

The proposed network outperforms state-of-the-art algorithms in depth and defocus estimation.

Real-image training data improves model robustness and accuracy.

Multi-task learning leverages physical connections for better feature learning.

Abstract

Monocular depth estimation and defocus estimation are two fundamental tasks in computer vision. Most existing methods treat depth estimation and defocus estimation as two separate tasks, ignoring the strong connection between them. In this work, we propose a multi-task learning network consisting of an encoder with two decoders to estimate the depth and defocus map from a single focused image. Through the multi-task network, the depth estimation facilitates the defocus estimation to get better results in the weak texture region and the defocus estimation facilitates the depth estimation by the strong physical connection between the two maps. We set up a dataset (named ALL-in-3D dataset) which is the first all-real image dataset consisting of 100K sets of all-in-focus images, focused images with focus depth, depth maps, and defocus maps. It enables the network to learn features and solid physical connections between the depth and real defocus images. Experiments demonstrate that the network learns more solid features from the real focused images than the synthetic focused images. Benefiting from this multi-task structure where different tasks facilitate each other, our depth and defocus estimations achieve significantly better performance than other state-of-art algorithms. The code and dataset will be publicly available at https://github.com/cubhe/MDDNet.