FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements

TL;DR

FlatNet is a deep learning framework that significantly improves the quality of scene reconstructions from lensless camera measurements, enabling photorealistic images without iterative optimization.

Contribution

The paper introduces FlatNet, a novel non-iterative deep learning method for high-quality lensless scene reconstruction, outperforming traditional iterative algorithms.

Findings

Orders of magnitude improvement in image quality.

Effective reconstruction on real scenes with different lensless prototypes.

Fast and photorealistic image generation.

Abstract

Lensless imaging has emerged as a potential solution towards realizing ultra-miniature cameras by eschewing the bulky lens in a traditional camera. Without a focusing lens, the lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, the current iterative-optimization-based reconstruction algorithms produce noisier and perceptually poorer images. In this work, we propose a non-iterative deep learning based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called $\textit{FlatNet}$, lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras, where the camera's forward model formulation is known. FlatNet consists of two stages: (1) an inversion stage that maps the measurement into a space of intermediate reconstruction by learning parameters within the forward model formulation, and (2) a perceptual enhancement stage that improves the perceptual quality of this intermediate reconstruction. These stages are trained together in an end-to-end manner. We show high-quality reconstructions by performing extensive experiments on real and challenging scenes using two different types of lensless prototypes: one which uses a separable forward model and another, which uses a more general non-separable cropped-convolution model. Our end-to-end approach is fast, produces photorealistic reconstructions, and is easy to adopt for other mask-based lensless cameras.