Endoscopic Depth Measurement and Super-Spectral-Resolution Imaging

TL;DR

This paper introduces an innovative endoscopic system that combines hyperspectral imaging, structured lighting, and neural network upscaling to provide real-time, dense surface shape and spectral information for surgical guidance.

Contribution

The work presents a novel integrated optical system that combines sparse hyperspectral data with structured lighting and neural networks for super-resolution, enabling real-time intraoperative surface and spectral measurements.

Findings

Real-time surface shape reconstruction with known scale.

Dense hyperspectral stacks generated from sparse data.

Validated system on ex vivo and in vivo experiments.

Abstract

Intra-operative measurements of tissue shape and multi/ hyperspectral information have the potential to provide surgical guidance and decision making support. We report an optical probe based system to combine sparse hyperspectral measurements and spectrally-encoded structured lighting (SL) for surface measurements. The system provides informative signals for navigation with a surgical interface. By rapidly switching between SL and white light (WL) modes, SL information is combined with structure-from-motion (SfM) from white light images, based on SURF feature detection and Lucas-Kanade (LK) optical flow to provide quasi-dense surface shape reconstruction with known scale in real-time. Furthermore, "super-spectral-resolution" was realized, whereby the RGB images and sparse hyperspectral data were integrated to recover dense pixel-level hyperspectral stacks, by using convolutional neural networks to upscale the wavelength dimension. Validation and demonstration of this system is reported on ex vivo/in vivo animal/ human experiments.