End-to-end Learning for Joint Depth and Image Reconstruction from Diffracted Rotation

TL;DR

This paper introduces an end-to-end deep learning method that jointly optimizes a phase mask and a neural network for improved monocular depth estimation from diffracted rotation, also incorporating image deblurring.

Contribution

It proposes a novel differentiable physical model for the aperture mask and a joint optimization framework for depth estimation and image deblurring, reducing complexity and data requirements.

Findings

Outperforms existing monocular depth estimation methods on indoor benchmarks.

Effectively recovers sharp images from RPSF-blurred inputs.

Requires less training data and model complexity.

Abstract

Monocular depth estimation is still an open challenge due to the ill-posed nature of the problem at hand. Deep learning based techniques have been extensively studied and proved capable of producing acceptable depth estimation accuracy even if the lack of meaningful and robust depth cues within single RGB input images severally limits their performance. Coded aperture-based methods using phase and amplitude masks encode strong depth cues within 2D images by means of depth-dependent Point Spread Functions (PSFs) at the price of a reduced image quality. In this paper, we propose a novel end-to-end learning approach for depth from diffracted rotation. A phase mask that produces a Rotating Point Spread Function (RPSF) as a function of defocus is jointly optimized with the weights of a depth estimation neural network. To this aim, we introduce a differentiable physical model of the aperture mask and exploit an accurate simulation of the camera imaging pipeline. Our approach requires a significantly less complex model and less training data, yet it is superior to existing methods in the task of monocular depth estimation on indoor benchmarks. In addition, we address the problem of image degradation by incorporating a non-blind and non-uniform image deblurring module to recover the sharp all-in-focus image from its RPSF-blurred counterpart.