Estimation of Tissue Oxygen Saturation from RGB images and Sparse Hyperspectral Signals based on Conditional Generative Adversarial Network

TL;DR

This paper introduces Dual2StO2, a dual-input cGAN model that accurately and rapidly estimates tissue oxygen saturation from RGB and sparse hyperspectral data, improving real-time intraoperative monitoring.

Contribution

The study presents a novel dual-input cGAN architecture that fuses RGB and sparse hyperspectral signals for efficient tissue oxygen saturation estimation, outperforming previous methods.

Findings

Dual2StO2 achieves higher accuracy than SSRNet.

The method is faster and more structurally accurate.

Performance improves with more fiber inputs.

Abstract

Purpose: Intra-operative measurement of tissue oxygen saturation (StO2) is important in the detection of ischemia, monitoring perfusion and identifying disease. Hyperspectral imaging (HSI) measures the optical reflectance spectrum of the tissue and uses this information to quantify its composition, including StO2. However, real-time monitoring is difficult due to the capture rate and data processing time. Methods: An endoscopic system based on a multi-fiber probe was previously developed to sparsely capture HSI data (sHSI). These were combined with RGB images, via a deep neural network, to generate high-resolution hypercubes and calculate StO2. To improve accuracy and processing speed, we propose a dual-input conditional generative adversarial network (cGAN), Dual2StO2, to directly estimate StO2 by fusing features from both RGB and sHSI. Results: Validation experiments were carried out on in vivo porcine bowel data, where the ground truth StO2 was generated from the HSI camera. The performance was also compared to our previous super-spectral-resolution network, SSRNet in terms of mean StO2 prediction accuracy and structural similarity metrics. Dual2StO2 was also tested using simulated probe data with varying fiber number. Conclusions: StO2 estimation by Dual2StO2 is visually closer to ground truth in general structure, achieves higher prediction accuracy and faster processing speed than SSRNet. Simulations showed that results improved when a greater number of fibers are used in the probe. Future work will include refinement of the network architecture, hardware optimization based on simulation results, and evaluation of the technique in clinical applications beyond StO2 estimation.

Figures

Tables

Table 1. Table 1 : The average SSIM and average mean prediction error of S ​ t ​ O 2 𝑆 𝑡 subscript 𝑂 2 StO_{2} with standard deviation estimation by Dual2StO2, SSRNet, and single input network with different sHSI parameters.

Fibre bundle Setting					Network	Results
$n_{spot}$	$\gamma$	$r$	$d$	Demo figure		Average SSIM	Average $\overline{e}$
300	0.25	2.6	10	Fig. 5(c)	Dual2StO2	$0.63\pm 0.17$	$0.11\pm 0.09$
					SSRNet	$0.54\pm 0.26$	$0.15\pm 0.12$
					Only sHSI	$0.54\pm 0.22$	$0.13\pm 0.11$
171	0.25	3.5	14	Fig. 5(b)	Dual2StO2	$0.61\pm 0.19$	$0.12\pm 0.1$
					SSRNet	$0.52\pm 0.24$	$0.17\pm 0.12$
					Only sHSI	$0.54\pm 0.22$	$0.15\pm 0.10$
121	0.25	4	16	Fig. 5(a)	Dual2StO2	$0.59\pm 0.21$	$0.14\pm 0.13$
					SSRNet	$0.54\pm 0.24$	$0.14\pm 0.10$
					Only sHSI	$0.53\pm 0.23$	$0.16\pm 0.10$
0	0	0	\	\	Dual2StO2	$0.53\pm 0.23$	$0.17\pm 0.14$
					SSRNet	$0.51\pm 0.24$	$0.18\pm 0.14$

Equations

$${\mathcal{L}}_{total}=$$

$$=$$

$$+\beta E_{s\sim p_{data(s)}}[||(f_{S\to T(s)})-t|{|}_1]\text{vspace}-1.0em$$

$$\mathcal{G}=\arg {\min }_G{\max }_D[{\mathcal{L}}_{cGAN}(G,D)+\beta {\mathcal{L}}_1(G)]\text{vspace}-1.0em$$

$$\bar{e}(i,j)=\frac{{\sum }_{i=0}^W{\sum }_{j=0}^{H}e(i,j)}{n_{effective}}=\frac{{\sum }_{i=0}^W{\sum }_{j=0}^H||I_{syn(i,j)}-I_{gt(i,j)}|{|}_1}{n_{effective}}$$

$$p_{HAP}=\frac{n_{HAP}}{n_{effective}}$$

Full text

Methods An endoscopic system based on a multi-fibre probe was previously developed to sparsely capture HSI data (sHSI). These were combined with RGB images, via a deep neural network, to generate high resolution hypercubes and calculate $StO_{2}$. To improve accuracy and processing speed, we propose a dual-input conditional generative adversarial network (cGAN), Dual2StO2, to directly estimate $StO_{2}$ by fusing features from both RGB and sHSI.

Results Validation experiments were carried out on in vivo porcine bowel data, where the ground truth $StO_{2}$ was generated from the HSI camera. Performance was also compared to our previous super-spectral-resolution network, SSRNet in terms of mean $StO_{2}$ prediction accuracy and structural similarity metrics. Dual2StO2 was also tested using simulated probe data with varying fibre number.

Conclusions $StO_{2}$ estimation by Dual2StO2 is visually closer to ground truth in general structure, achieves higher prediction accuracy and faster processing speed than SSRNet. Simulations showed that results improved when a greater number of fibres are used in the probe. Future work will include refinement of the network architecture, hardware optimisation based on simulation results, and evaluation of the technique in clinical applications beyond $StO_{2}$ estimation.

1 Introduction

Tissue perfusion and oxygenation are important clinical indicators of organ health during minimal access surgery (MAS). Endoscopic hyperspectral imaging (HSI) is a non-invasive optical technique to capture quantitative spectral information with a high spatial resolution based on narrow spectral bands over a virtually continuous spectral range for live tissue diagnostics and monitoring Lu and Fei (2014). HSI can be used to estimate oxygen saturation ($StO_{2}$) and perfusion, which reflects tissue function and the health of organ’s blood supply. This, in turn, can be applied to various important clinical applications Lu and Fei (2014), including monitoring of cortical haemodynamics during brain surgery Mori et al. (2014), reperfusion during organ transplantation Clancy et al. (2016) and detection of intestinal ischaemia Akbari et al. (2010). High resolution spectral data can also be used to characterise tissue and detect subtle differences between normal and dysplastic areas Kumashiro et al. (2016). HSI is a non-contact technique, compatible with conventional surgical light sources and endoscopes, and has some important advantages over competing optical techniques, such as photoacoustic tomography (PAT) Wang and Hu (2012), which requires ultrasound contact and a complex laser source.

HSI requires acquisition of a hypercube, which has one spectral and two spatial dimensions. Imaging hardware may use tuneable filters or spatial scanning, but does not typically achieve real-time operation due to the data capture and processing times. Snapshot spectral imaging acquires the entire hypercube simultaneously, but the number of wavelengths or spatial resolution must be sacrificed to achieve high speed acquisition. This trade-off between spectral information, spatial resolution and acquisition speed is a barrier for clinical use of HSI and other optical imaging techniques Lu and Fei (2014).

To overcome this, we previously developed a dual mode structured light and hyperspectral imaging (SLHSI) system Clancy et al. (2011)Lin et al. (2018) to capture sparse hyperspectral images in real-time, as illustrated in Fig. 1(a). The light (i.e. reflectance or fluorescence) from the tissue surface was imaged onto the 2D fibre array, and the bundle randomly re-ordered the fibres into a linear array at the other end. The spectrum carried by each fibre could then be captured by imaging the linear array onto the entrance slit of an imaging spectrograph. The data could then be rearranged computationally to generate sparse hyperspectral images (sHSI) in a snapshot. The 2.8 mm fibre bundle can be inserted through an endoscope biopsy port or attached to the endoscope or another surgical instrument Clancy et al. (2011). The system could also be used to record spectrally-encoded structured lighting (SL) images Clancy et al. (2011)Lin et al. (2018), although this capability is not explored further in this paper.

To process the acquisition, a super-spectral-resolution network, called SSRNet, was proposed to integrate dense RGB images and sHSI for pixel-level hypercube estimation Lin et al. (2018). The hypercube could be used to estimate $StO_{2}$ based on the modified Beer-Lambert law as illustrated in processing Route 1 (Blue line) in Fig. 1(b). Previous work also explored the feasibility of estimating $StO_{2}$ directly from RGB images (Route 2, Fig. 1(b)) Li et al. (2018), and showed that hyperspectral information improves the accuracy of the result Lin et al. (2017). However, as the aim of SSRNet was to predict dense HSI hypercubes, it was not explicitly optimised for $StO_{2}$ estimation, and the value of combining RGB images with sparse HSI hypercubes has not been evaluated for estimating dense $StO_{2}$ maps.

In this paper, we extend the previous published results, proposing a dual input network using cGAN, Dual2StO2, to achieve dense $StO_{2}$ estimation using end-to-end learning, without the need for the intermediate spectral estimation step. The proposed network was inspired by the performance of GANs in super-resolution Ledig et al. (2017)Yi et al. (2018), to achieve super-resolution estimation in spatial (for sHSI) and spectral (for RGB) domains. A minimax two-player game was utilised between the generator and discriminator to further improve the accuracy of per-pixel regression for $StO_{2}$ estimation. By adding conditional input, the generator in cGAN would estimate StO2 imitating the structure of the condition, instead of random image generation in GANs. The results from Mirza et al. Mirza and Osindero (2014) and Isola et al. Isola et al. (2017) also supported that cGAN could achieve higher pixel-level accuracy than other GANs with the same settings. The relationship between two input modalities (RGB, sHSI) and output (StO2) were known a priori, which enabled the network to be trained by supervised learning, achieving faster convergence and prediction accuracy. Additionally, a customized mask was added to filter saturated pixels and unreliable estimates at the pixel level. Furthermore, one of the key parameters in designing the MSI data acquisition system is the number of fibres in the bundle, and we have therefore additionally simulated the performance of this estimation for different fibre bundles. This approach is represented by Route 3 (Purple line) in Fig. 1(b). This will allow optimisation of future hardware designs to increase robustness.

In this paper, Sec. 2.1 will describe data acquisition and HSI data synthesis, while Dual2StO2 is presented in Sec. 2.2. The evaluation metrics and validation setup for this method are described in Sec. 3, followed by a validation of the network via an inter-animal study on porcine bowel in vivo. The previous two-stage $StO_{2}$ estimation approach (Route 1, Blue line) developed by Lin et al. Lin et al. (2017) in Fig. 1(b) was adopted as the baseline against which the performance of the proposed network was evaluated.

2 Materials and methods

2.1 Data Acquisition and Preprocessing

The porcine bowel in vivo data was captured by a liquid crystal tuneable filter (LCTF) in the range of $460-700\,\mathrm{nm}$ with $10\,\mathrm{nm}$ interval, as described in a previous work Clancy et al. (2015). Here, a subset of the spectral data from $460-690\,\mathrm{nm}$ was considered as a ground truth 24-channel hypercube with spatial size $256\times 192$ pixels. A total of 50 acquisitions were selected from 15 separate animals. Simulated RGB images. The RGB image (Input - x) was simulated from the hypercube (Route 3 in Fig. 1(b)) using the known spectral response of a colour camera Clancy et al. (2015) Clancy et al. (2012).

Analytical method to estimate $StO_{2}$. A well-established linear model based on modified Beer-Lambert law was used in this paper to obtain ground truth $StO_{2}$. It uses linear regression to estimate the relative concentrations of oxygenated and deoxygenated haemoglobin ($HbO_{2}$ and $Hb$), and calculates $StO_{2}$ as the quantity of $HbO_{2}$ as a fraction of total haemoglobin ($HbO_{2}+Hb$), subject to assumptions Clancy et al. (2015). Experimental validation has also been carried out in our previous in vivo uterine transplantation and bowel surgery experiments Clancy et al. (2015)Clancy et al. (2016) as well as by others Mori et al. (2014) Sorg et al. (2005) The coefficient of determination (CoD) Tjur (2009) was used to evaluate the accuracy of the linear regression estimation. CoD $\leq 0.85$ is set as threshold for linear regression outliers, and related pixels were excluded for training and evaluation. Pixels located in non-tissue regions, insufficiently illuminated area and specular reflections were also excluded.

Synthesized sparse hyper-spectral images. A number of different distal tip fibre arrangements may be chosen for the experimental hardware. To study how this may affect the performance of the Dual2StO2 network and thereby influence future decisions on the experimental setup, we have simulated data acquired with different fibre arrangements from the dense hyperspectral dataset in Sec. 2.1. The use of circular masks with high resolution ground truth images to simulate and assess the performance of imaging fibre bundles has previously been demonstrated Kyrish et al. (2010) Shao et al. (2018). Masks were created in MATLAB (R2018a; The Math Works, Inc., USA) using a circular sensing area arranged on a hexagonal grid to represent the array of fibres, with the spatial information averaged within these areas, as illustrated in Fig. 2 and described in the following steps.

1. Step 1

Define a radius ($r$, representing the transmissive fibre cores), and horizontal and vertical spacing between the spot centres ($d$, a metric representing the core separations), where the ratio $\gamma=\frac{r}{d}$ is the fill factor that define the relationship between the area of projected spot and the space. In real situation, $\gamma=$ stays unchanged when changing fibre numbers, as the dimensions of individual fibres and their cladding are consistent, which also described in Table. 1; 2. Step 2

Generate a hexagonal grid across the whole image to simulate a hexagonal packed fibre probe ([$W_{s}$,$H_{s}$,$W_{e}$,$H_{e}$] = [$0,0,W,H$], where $W$ and $H$ are the image width and height, the subscripts $s$ and $e$ stands for start and end of the image range respectively); 3. Step 3

Define the radius ($R=\frac{h}{2}$) of the fibre bundle (green circle); 4. Step 4

Generate a mask that includes all fibre cores within the bundle; 5. Step 5

Finally, average the spatial information within each fibre sensing area to generate a single spectrum for each fibre.

Data Augmentation. The training data was augmented by horizontal, and vertical flipping, and image cropping using a sliding $96\times 96$ window with a stride of 16. The cropped images were resized to the target size, $256\times 256$ through bilinear interpolation, which augments 231 times of 38 original images of and results in 8778 images for training. In order to maintain information consistency, the central $96\times 96$ region cropped from original 12 images were resized into a $256\times 256$ through bilinear interpolation and used as test set.

2.2 Dual-input network for StO2 estimation

Dual2StO2 is a cGAN-based image-to-image translation network for $StO_{2}$ estimation utilizing dual input modalities (RGB, sHSI), which was implemented by Pytorch 4.0. In analogy with automatic language translation, image-to-image translation defined by Isola et al. Isola et al. (2017) is a task that translates the representation of one scene into another, which implemented as a general framework called pix2pix for per-pixel classification and regression. Its fundamental network was based on cGAN, where additional conditions were added for both the generator and discriminator Mirza and Osindero (2014).

Generator(G). Inspired by the network architecture of pix2pix Isola et al. (2017), it was adopted as the base model in the generator of Dual2StO2, because the relationship between two input modalities (RGB, sHSI) and output ($StO_{2}$) were known a priori (suitable for supervised learning). The network architecture of the generator was modified based on a multi-input unsupervised learning image-to-image translation framework, called In2I Perera et al. (2018), as illustrated in Fig. 3.

• •

The encoder (light orange box) was designed to first extract features from the RGB image ($256\times 256\times 3$) and sparse HSI ($256\times 256\times 24$) (pink region), fuse the feature map from these two modalities by concatenation (grey box), and extract further features from the fused feature map;

• •

The decoder (light green box) was introduced to decode the feature map and output the $StO_{2}$ estimation;

• •

Residual block (brown arrow, with process illustrated in the brown box) proposed by He et al. He et al. (2016) was adopted in both encoder and decoder;

• •

Instance normalization was adopted based on comparison work Ulyanov et al. (2016) Zhou and Yang (2019), where the results indicated that instance normalization has better performance in image generation tasks than batch normalization;

• •

One mask was created to filter the position of pixels with saturated pixel values due to specular reflections (NaN), and those with a coefficient of determination (CoD) $\leq 0.85$;

In the training stage of the simulation experiment, the two input modalities (simulated RGB, and the synthesized sparse hyper-spectral image called synthesized sHSI) defined as $S=\{S_{RGB},S_{sHSI}\}$ were fed into the generator in Fig. 3 and trained to learn a forward transformation $f_{S\rightarrow T(s)}$ to output a single set of images ($StO_{2}$) from Sec. 2.1, under the condition of source domain $S$. Here, the source and target domain were defined by $S$ and $T$, with the data distributions of domain $S$ and $T$ as $p_{data(s)}$ and $p_{data(t)}$. The similar notations in In2I Perera et al. (2018) are used here. Discriminator(D). Under the condition of observed image ($\mathbf{x}$) from input domain $S$, the discriminator D will estimate the probability of whether the image is the ground truth image ($\mathbf{y}$) from target domain $T$, or the synthesized image ($f_{S\rightarrow T(s)},\mathbf{\hat{y}}$) generated by generator G. A convolutional network, called “PatchGAN”, was first introduced by Li et al. Li and Wand (2016) to classify real or fake images based on individual image patches. A comparison on different patch size was carried by Isola et al. Isola et al. (2017) and showed that the performance of image-to-image translation were best with $70\times 70$ patch size. This size was adopted into the implementation of the discriminator. Concat $(\mathbf{x},\mathbf{y})$ and Concat $(\mathbf{x},\mathbf{\hat{y}})$ are put into a discriminator separately, which outputs the probability of the input to be $\mathbf{y}$. Here Concat() is concatenate, the probability map is a $30\times 30\times 1$ map which is useful for pixel-level rather than image-level translation. The discriminator network architecture is shown in Fig. 4.

Adversarial learning. During the training stage, the generator tries to generate a synthesised image ($f_{S\rightarrow T(s)}$) as real as possible to cheat the discriminator to consider it as real. On the other hand, the discriminator will also improve its ability to make correct judgement on whether the images are the ground truth image from target domain $T$, or the synthesized image ($f_{S\rightarrow T(s)}$) generated by generator. Hence, this forward transformation ($f_{S\rightarrow T(s)}$) is trained by adversarial loss fhe unction Eq. 2.2:

[TABLE]

where D is the discriminator and $\beta$ is the weight of the L1 norm, set as 400 based on previous work Li et al. (2018).

A minimax two-player game is introduced in this network to train the generator and discriminator through an adversarial process. Hence, the generator in Fig. 3 is trained to maximize the probability of discriminator to be a false positive. Our final objective is defined by Eq. 2:

[TABLE]

where $G$ is the generator.

3 Experiments

In this section, the evaluation metrics are firstly defined in the Sec. 3.1 to quantitatively analyse the performance of $StO_{2}$ estimation, followed by the experiment setup in the Sec. 3.2.

3.1 Evaluation metrics.

• •

Structural similarity index (SSIM): a perception-based method proposed by Wang et al.. Wang et al. (2004) comparing the local patterns of pixel intensities that have been normalized for luminance and contrast. Tlshe similarity between the ground truth and synthesized image could be measured and quantified between [math] and $1$, where $SSIM=1$ was considered as identical.

• •

Mean prediction error ($\bar{e}$): the difference in $StO_{2}$ value between the ground truth and synthesized image at each pixel measured by the L1 norm.

[TABLE]

where $I_{syn}$ and $I_{gt}$ are the absolute values for a pixel at column $i$, row $j$, in the synthesized and ground truth images with width $W$ and height $H$, respectively, and $n_{effective}$ is the total number of pixels in the image excluding saturated and low CoD pixels

• •

Fraction of pixels with high accuracy level ($p_{HAP}$): accuracy is defined above a certain level compared to the pixel data in the ground truth image.

[TABLE]

where $n_{HAP}$ is the number of pixels with high prediction accuracy (i.e. $1-e_{(i,j)}\geq 95\%$), .

3.2 Experimental set-up.

Animal studies were carried out to validate the performance of Dual2StO2 on the in vivo acquisitions by separating animals into training and test data sets. The training set consisted of 38 acquisitions captured for 10 animals (animal ID: 1 to 10), while 12 further test acquisitions were from the 5 remaining animals (animal ID: 11 to 15).

The bundles with circular distal cross sections and different numbers of fibres ($n_{spot}=0,121,171,300$) were simulated. Two of these configurations (121 and 171 spots) were chosen as they match the existing hardware available, complemented by a fibre bundle with a high number of spots ($300$). To confirm any benefit of integrating sparse HSI, a fibre bundle with zero spots was used as the control group. Fig. 5 illustrates sample synthesized sHSI images generated by the corresponding masks. These sHSI and simulated RGB images were fed into Dual2StO2. Route 1 in Fig. 1(b) based on the SSRNet developed by Lin et al., was adopted as the baseline to compare the performance of $StO_{2}$ based on the same simulated RGB and synthesized sHSI as input.

4 Results.

Table. 1 summarises the performance of Dual2StO2 and SSRNet compared to the ground truth. The proposed network is superior to SSRNet in terms of SSIM and pixel-level accuracy across all fibre bundle configurations. When the fill factor ($\gamma$) is unchanged, the predicted images by Dual2StO2 are structurally closer to the ground truth ($16.5\%$ higher average SSIM), and have $3.6\%$ lower average $\bar{e}$ than SSRNet for $n_{spot}=300$. Fig. 6(a) shows that even when the number of fibres increased in the bundle the Dual2StO2 predicted images are still structurally closer to the ground truth, with higher SSIM and less variance across different animals indicated by smaller interquartile range (IQR) at high SSIM value than SSRNet. Fig. 6(b) and Fig. 6(c) also presented lower $\bar{e}$ and larger, $p_{HAP}$ by Dual2StO2 than that by SSRNet. Faster $StO_{2}$ estimation ($\approx 35\,\mathrm{ms}$) can be achieved by Dual2StO2 due to its the end-to-end estimation without the intermediate spectral estimation step and light-weight architecture, while the estimation required over $500\,\mathrm{ms}$ by SSRNet Lin et al. (2018). This was validated on a PC (OS: Ubuntu 16.04; processor: i7-3770; graphics card: NVIDIA GTX TITAN X).

A better $StO_{2}$ estimation was achieved with a higher number of fibres in the bundle with $n_{spot}=300$ achieving the best result for both Dual2StO2 and SSRNet. The overall performance of $StO_{2}$ estimation was better with additional sHSI information than when compared to that from RGB images only. When sHSI was added, comparing $n_{spot}=121$ to $n_{spot}=0$, the structural similarity increased by $10\%$ and the average mean error reduced by $2.3\%$. An experiment has also been carried out to estimate $StO_{2}$ using only sHSI. The results from Table. 1 indicate that the single input network can estimate $StO_{2}$ and achieve pixel-level accuracy, evaluated by averaged mean prediction error, to some extent. However, the general structure similarity, evaluated by SSIM, is lower than that the dual input network combined with RGB images. As the number of spots increased the performance of $StO_{2}$ estimation by single input network also improved.

Fig. 7 and Fig. 8 illustrate the typical performance of $StO_{2}$ estimation by Dual2StO2 and SSRNet with the number of fibres in the bundle ($n_{spot}=300$) on the second acquisition in animal ID 13 (porcine bowel). The input RGB image, reference $StO_{2}$ and estimated $StO_{2}$ by Dual2StO2 and SSRNet are displayed, while the $StO_{2}$ value difference between them is also presented. These demonstrate that, with an end-to-end learning training/testing architecture, Dual2StO2 outperforms the two-stage method, i.e. estimating $StO_{2}$ from hypercubes generated by SSRNet.

5 Discussion and conclusions

A dual input network, called Dual2StO2, was designed to estimate $StO_{2}$ based on sHSI and RGB images. Simulations of three fibre bundles ($n_{spot}=121$, $171$, $300$) and a control group ($n_{spot}=0$) were carried out to investigate the impact of integrating sHSI, and to examine the relationship between the number of fibres and prediction accuracy. The results showed that with same fibre bundle, Dual2StO2 has better performance in $StO_{2}$ estimation (higher SSIM and lower $\bar{e}$ with smaller IQR, larger $p_{HAP}$ and faster prediction) than SSRNet. Compared with the control group ($n_{spot}=0$, using RGB data alone), the simulation results showed that the overall performance of $StO_{2}$ estimation with both Dual2StO2 and SSRNet was improved by adding sHSI. Performance was also improved as the number of fibres increased from $121$ to $300$, in terms of prediction accuracy and structural similarity. The result of the control group also indicated that $StO_{2}$ can be estimated directly from RGB although with consistently lower accuracy. This is in agreement with our previous works Li et al. (2018) Lin et al. (2017). It was also observed that although RGB data could produce realistic spectral estimation, large errors at individual wavelengths were common. While StO2 estimation may be relatively insensitive to these underlying errors, spectral fidelity will be crucial to solving more subtle diagnostic problems such as the detection of cancer. This will be explored further in our future work.

For real fibre bundles, the transmission characteristics of each fibre differs and cross-talk between fibres may result in measurement noise. This does not affect the result of the Dual2StO2 versus SSRNet comparison, but will affect the spatial accuracy of the sHSI-only results in Table. 1 although it is unlikely to be significant. Furthermore, the sHSI presented here is simulated from an LCTF-based hyperspectral camera, which has lower spectral resolution (10-20 nm) than the spectrograph used in the real SLHSI system ($\approx 5$ nm). Therefore, it is likely that the overall StO2 accuracy would be improved when trained with data from the real SLHSI bundle. Nevertheless, the simulations presented here serve as a useful testbed to allow comparative testing of network performance and to guide future design of an optimised fibre bundle. The network architecture of Dual2StO2 will be further customized for better performance, including exploration of custom-designed networks to extract features from RGB and sHSI images separately. The proposed dual input network could potentially be modified to achieve dual output and generate, for example, narrow band images (NBI). The pyramid architecture of multi-generator and discriminator proposed by Wang et al. Wang et al. (2018a) could also be adopted to enhance the quality of generation. Our network can be further extended to real-time $StO_{2}$ imaging based on video-to-video synthesis Wang et al. (2018b).

Acknowledgements.

The authors appreciate the assistance of Xiao-Yun Zhou on the GPU configuration and academic advice, and help from Maria Leiloglou. We would like to thank NVIDIA Corporation for the donation of the Titan X GPU. This work was carried out with support from the CRUK Imperial Centre, Imperial ECMC and the NIHR Imperial BRC.

**Compliance with ethical standards

Conflict of interest** The authors declare that they have no conflict of interest.

Ethical standards This article does not contain any studies with human participants. All applicable international, national and/or institutional guidelines for the care and use of animals were followed.