Robust super-resolution depth imaging via a multi-feature fusion deep network

TL;DR

This paper introduces a deep learning method that fuses multiple features from SPAD camera data to enhance the resolution and denoise depth images without additional sensors, benefiting real-time 3D imaging applications.

Contribution

A novel multi-feature fusion deep network designed specifically for SPAD camera data to improve depth image resolution and denoising without extra sensors.

Findings

Achieves four-fold depth resolution enhancement.

Provides effective denoising across various noise levels.

Demonstrates significant improvement in 3D imaging quality.

Abstract

Three-dimensional imaging plays an important role in imaging applications where it is necessary to record depth. The number of applications that use depth imaging is increasing rapidly, and examples include self-driving autonomous vehicles and auto-focus assist on smartphone cameras. Light detection and ranging (LIDAR) via single-photon sensitive detector (SPAD) arrays is an emerging technology that enables the acquisition of depth images at high frame rates. However, the spatial resolution of this technology is typically low in comparison to the intensity images recorded by conventional cameras. To increase the native resolution of depth images from a SPAD camera, we develop a deep network built specifically to take advantage of the multiple features that can be extracted from a camera's histogram data. The network is designed for a SPAD camera operating in a dual-mode such that it captures alternate low resolution depth and high resolution intensity images at high frame rates, thus the system does not require any additional sensor to provide intensity images. The network then uses the intensity images and multiple features extracted from downsampled histograms to guide the upsampling of the depth. Our network provides significant image resolution enhancement and image denoising across a wide range of signal-to-noise ratios and photon levels. We apply the network to a range of 3D data, demonstrating denoising and a four-fold resolution enhancement of depth.