A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging

TL;DR

This paper introduces a self-supervised and adversarial method for hyperspectral demosaicking and RGB reconstruction in surgical imaging, enabling real-time spectral imaging without requiring paired high-resolution datasets.

Contribution

It presents a novel self-supervised approach that leverages unpaired surgical images and adversarial learning to improve hyperspectral demosaicking and RGB reconstruction in intra-operative settings.

Findings

Improved spatial and spectral fidelity of reconstructed images.

Enhanced RGB visualization quality confirmed by neurosurgical experts.

Outperforms existing demosaicking methods in quantitative metrics.

Abstract

Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. However, snapshot mosaic imaging requires a demosaicking algorithm to fully restore the spatial and spectral details in the images. Modern demosaicking approaches typically rely on synthetic datasets to develop supervised learning methods, as it is practically impossible to simultaneously capture both snapshot and high-resolution spectral images of the exact same surgical scene. In this work, we present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth. We leverage unpaired standard high-resolution surgical microscopy images, which only provide RGB data but can be collected during routine surgeries. Adversarial learning complemented by self-supervised approaches are used to drive our hyperspectral-based RGB reconstruction into resembling surgical microscopy images and increasing the spatial resolution of our demosaicking. The spatial and spectral fidelity of the reconstructed hyperspectral images have been evaluated quantitatively. Moreover, a user study was conducted to evaluate the RGB visualisation generated from these spectral images. Both spatial detail and colour accuracy were assessed by neurosurgical experts. Our proposed self-supervised demosaicking method demonstrates improved results compared to existing methods, demonstrating its potential for seamless integration into intra-operative workflows.