Deep Hyperspectral-Depth Reconstruction Using Single Color-Dot Projection

TL;DR

This paper introduces a novel single-shot method for simultaneous depth and hyperspectral reflectance reconstruction using a simple RGB camera and projector, leveraging a color-dot pattern and an end-to-end neural network.

Contribution

The proposed approach uniquely combines depth and spectral reconstruction in one shot without special hardware, using a single color-dot projection and a specialized neural network architecture.

Findings

Outperforms existing separate hyperspectral and depth reconstruction methods

Uses a simple off-the-shelf RGB camera and projector setup

Demonstrates effective joint reconstruction in experiments

Abstract

Depth reconstruction and hyperspectral reflectance reconstruction are two active research topics in computer vision and image processing. Conventionally, these two topics have been studied separately using independent imaging setups and there is no existing method which can acquire depth and spectral reflectance simultaneously in one shot without using special hardware. In this paper, we propose a novel single-shot hyperspectral-depth reconstruction method using an off-the-shelf RGB camera and projector. Our method is based on a single color-dot projection, which simultaneously acts as structured light for depth reconstruction and spatially-varying color illuminations for hyperspectral reflectance reconstruction. To jointly reconstruct the depth and the hyperspectral reflectance from a single color-dot image, we propose a novel end-to-end network architecture that effectively incorporates a geometric color-dot pattern loss and a photometric hyperspectral reflectance loss. Through the experiments, we demonstrate that our hyperspectral-depth reconstruction method outperforms the combination of an existing state-of-the-art single-shot hyperspectral reflectance reconstruction method and depth reconstruction method.