Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks

TL;DR

This paper introduces an uncertainty-aware framework using invertible neural networks to assess optical imaging modalities, providing detailed posterior distributions for tissue parameter estimation and guiding hardware optimization.

Contribution

It presents a novel invertible neural network-based method for comprehensive uncertainty assessment of optical imaging hardware, capturing ambiguity in tissue parameter inference.

Findings

Oxygenation estimation is well-posed.

Blood volume fraction estimation may be ambiguous.

Increasing spectral bands reduces ambiguity.

Abstract

Purpose: Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed. Methods: We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors. Results: Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) Estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera. Conclusion: Our method could help to optimize optical camera design in an application-specific manner.

Tables

Table 1. Table 1 : The physiological parameter ranges used for simulating the desired tissue model.

layer	$v_{\text{HB}}[\%]$	$s{O}_2[\%]$	$a_{\text{mie}}\left[{\mathrm{cm}}^{-1}\right]$	$b$	$g$	$n$	$d\left[\mathrm{mm}\right]$
1	0-10	0-100	$18.9\pm 10.2$	1.286	0.8-0.95	1.33	0.06-0.100
2	0-10	0-100	$18.9\pm 10.2$	1.286	0.8-0.95	1.36	0.06-0.085
3	0-10	0-100	$18.9\pm 10.2$	1.286	0.8-0.95	1.38	0.04-0.060
${\mu }_a(v_{\text{HB}},s,\lambda )=v_{\text{HB}}\left(s{ϵ}_{\text{HBO2}}\left(\lambda \right)+\left(1-s\right){ϵ}_{\text{HB}}\left(\lambda \right)\right)\ln \left(10\right)\left(150g{L}^{-1}\right){(64,500g{\mathrm{mol}}^{-1})}^{-1}$
${\mu }_s(a_{\text{mie}},b,\lambda )=\frac{a_{\text{mie}}}{1-g}{\left(\frac{\lambda }{500\mathrm{nm}}\right)}^{-b}$
framework: MCML[1], ${10}^6$ photons per simulation
wavelength range $\left(\Lambda \right)$: 450-$720\mathrm{nm}$ (stepsize=$2\mathrm{nm})$

Table 2. Table 2 : Results for the comparative camera assessment. MM: Percentage of multi-modal posteriors; RMSE: Root-mean-square error; W: Median width of the 68% symmetric confidence interval; pp: percentage points; The value in brackets denotes the percentage of samples with W ≥ 0.5 ⋅ σ p ​ r ​ i ​ o ​ r 𝑊 ⋅ 0.5 subscript 𝜎 𝑝 𝑟 𝑖 𝑜 𝑟 W\geq 0.5\cdot\sigma_{prior} , where σ p ​ r ​ i ​ o ​ r subscript 𝜎 𝑝 𝑟 𝑖 𝑜 𝑟 \sigma_{prior} represents the standard deviation of corresponding the prior distribution. Note that a prerequisite for a distribution to be classified as multi-modal is W < 0.5 ⋅ σ p ​ r ​ i ​ o ​ r 𝑊 ⋅ 0.5 subscript 𝜎 𝑝 𝑟 𝑖 𝑜 𝑟 W<0.5\cdot\sigma_{prior} .

Camera	$s{O}_2$			$v_{\text{HB}}$
	MM [%]	RMSE [pp]	W [pp]	MM [%]	RMSE [pp]	W [pp]
27-equi	0.3	2.3	2.3 (0%)	0.1	1.6	3.4 (67%)
8-med	0.5	2.9	4.0 (0%)	0.0	1.7	3.5 (62%)
3-nRGB	9.3	4.8	5.8 (0%)	0.0	1.7	3.1 (64%)
3-med	3.6	5.7	8.6 (0%)	0.0	2.4	5.3 (99%)

Equations

$$r_i=\frac{{\int }_{{\lambda }_{\text{min}}}^{{\lambda }_{\text{max}}}o(\lambda )l(\lambda )f_i(\lambda )r(\lambda )\text{d}\lambda }{{\int }_{{\lambda }_{\text{min}}}^{{\lambda }_{\text{max}}}o(\lambda )l(\lambda )f_i(\lambda )\text{d}\lambda }$$

$$\mathbf{x}=g(\mathbf{y},\mathbf{z};\Theta )\text{ with }\mathbf{z}\sim p(\mathbf{z})=\mathcal{N}(\mathbf{z};0,I),$$

Full text

Methods: We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors.

Results: Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) Estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera.

Conclusion: Our method could help to optimize optical camera design in an application-specific manner.

Conflict of Interest: The authors declare that they have no conflict of interest.

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. All applicable international, national, and/or institutional guidelines for the care and use of animals were followed. All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were conducted. For this type of study formal consent is not required.

1 Introduction

Many key challenges in the intersection of natural sciences and the life sciences are related to solving inverse problems. Here, it is assumed that a forward process maps the (hidden) parameters of interest $\mathbf{x}\in X$ to observations $\mathbf{y}\in Y$ that can be measured. In the context of computer assisted interventions (CAI), for example, $\mathbf{x}$ may refer to important physiological tissue parameters, such as the tissue oxygenation $sO_{2}$ (cf. Figure 1), while $\mathbf{y}$ may represent multispectral measurements of tissue. The problem is usually solved by regression, which gives a point estimate for the tissue parameter(s) of interest based on the camera measurements [3, 30, 29]. However, in most inverse problems the mapping between $X$ and $Y$ is not injective, and two substantially different $\mathbf{x_{1}},\mathbf{x_{2}}\in X$ can result in the same $\mathbf{y}$. To recover a unique inverse, a regularizer can be added to the objective, but this approach, although commonly used, neglects the inherent ambiguity of the solution. For our application, an explicit analysis of the ambiguity is crucial to identify the most suitable camera in terms of number and characteristics of camera bands. To our knowledge, none of the existing parameter estimation methods has incorporated a sufficiently powerful uncertainty quantification to do so.

Current approaches to uncertainty quantification in the field of deep learning, such as dropout sampling (cf. e.g. [7, 16, 17, 24]), probabilistic inference (cf. e.g. [6, 14, 31]) or ensembles of estimators (cf. e.g. [15, 23]), typically augment traditional point estimates with confidence intervals, but do not recover unrestricted full posteriors $p(\mathbf{x}|\mathbf{y})$. Consequently, these methods do not account for the possibility that the same observation $\mathbf{y}$ corresponds to fundamentally different $\mathbf{x}$. In other words these methods would always assume that $\mathbf{x}$ follows the blue (unimodal) distribution depicted in Figure 2(a) even if $\mathbf{x}$ followed the orange (multimodal) distribution. The following two cases illustrate that this is a serious shortcoming when we wish to recover a physiological parameter $\mathbf{x}$ from observations $\mathbf{y}$ (Figure 2(b)):

The solution is unique but suffers from high uncertainty. This may be represented by a uni-modal posterior $p(\mathbf{x}|\mathbf{y})$ whose single mode has large standard deviation. 2. 2.

The problem is ill-posed in the sense that two substantially different $\mathbf{x_{1}},\mathbf{x_{2}}$ yield the same $\mathbf{y}=f(\mathbf{x_{1}})=f(\mathbf{x_{2}})$. This must be represented by a multi-modal posterior $p(\mathbf{x}|\mathbf{y})$ whose individual modes may have low uncertainty.

Forcing a uni-modal representation onto the second case cannot work: It would either focus on one of the modes and miss the other, or cover both solutions under a single wide mode (similar to case 1) whose maximum is located at the average of $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ – a highly implausible $\mathbf{x}$-value for the given $\mathbf{y}$.

We therefore argue that an ideal method for comparative camera assessment should be able to deal with all possible types of uncertainty. We propose to move beyond point estimates by mapping an observation $\mathbf{y}$ to a full distribution $p(\mathbf{x}\,|\,\mathbf{y})$ rather than a single $\mathbf{x}$. To this end, we solve the resulting inverse problem using the recently proposed concept of invertible neural networks (INNs) [2]. Performance measures for a hardware setup can then be computed from the number and widths of the modes of the posteriors, as illustrated in Figure 2(b).

In the following sections, we describe our approach in detail and apply it to the comparative assessment of four different camera designs given the specific use case of physiological parameter estimation from multispectral imaging data.

2 Methods

In this section, we formalize the proposed approach to performance assessment in a generic manner and apply it to the specific use case of camera selection for multispectral image analysis.

Generally speaking, we assume that the method to be assessed involves a hardware setup $H$ (e. g. a multispectral camera) that is used to solve an inverse problem with a well-known forward process $f:X\to Y$, such as the mapping of tissue oxygenation $\mathbf{x}$ to the pixel-wise measurement $\mathbf{y}=f(\mathbf{x})$ of a multispectral camera. We further assume that we have access to a data set $T=T^{\text{train}}\cup T^{\text{validation}}\cup T^{\text{test}}$ composed of tuples $(\mathbf{x},\mathbf{y})$, with $\mathbf{y}=f(\mathbf{x})$. Typically $T$ can be generated by means of Monte Carlo simulation, as in [13, 29, 30] assuming the (virtual) hardware setup $H$. Finally, we represent the regressor $r$ as an invertible neural network, as detailed in Section 2.2.

Our approach to performance assessment involves the following steps: (1) Training the regressor $r$ on $T^{\text{train}}$ using $T^{\text{validation}}$ for hyperparameter tuning. (2) Applying $r$ to $T^{\text{test}}$, to get a target distribution $p(\mathbf{x}\,|\,\mathbf{y})$ for each $\mathbf{y}$ in the test data set. (3) Extracting the modes for each $p(\mathbf{x}\,|\,\mathbf{y})$. (4) Computing descriptive statistics over the number and widths of the modes to quantify the uncertainty of the regressor. Different hardware setups can then be compared using metrics that consider not only the accuracy but also the uncertainty characteristics of the regressor. The following paragraphs instantiate this approach in the specific context of camera selection for intra-operative physiological parameter estimation.

2.1 Data generation for performance assessment

We apply Monte Carlo methods to generate tuples of physiological parameters $\mathbf{x}$ and corresponding pixel-wise measurements $\mathbf{y}=f(\mathbf{x})$. The method is based on previous work [30] and briefly revisited here.

Tissue is assumed to be composed of three infinitely wide layers. Each layer is defined by the following tissue parameters: blood volume fraction $v_{\text{HB}}$, reduced scattering coefficient at 500nm $a_{\text{mie}}$, scattering power $b_{\text{mie}}$, anisotropy $g$, refractive index $n$ and layer thickness $d$. Based on values for hemoglobin extinction coefficients $\epsilon_{\text{HB}}$ and $\epsilon_{\text{HBO2}}$ from literature [11], absorption and scattering coefficients $\mu_{a}$ and $\mu_{s}$ have been determined for use in the MC simulation framework. A Graphics Processing Unit (GPU) accelerated version [1] of the Monte Carlo Multi-Layered (MCML) simulation framework [26] was chosen to generate spectral reflectances. The spectral reflectances $r(\lambda)$ as determined by the MC simulation can be transformed to the reflectance measurement $r_{i}$ at band $i$ of a given camera $c$ by:

[TABLE]

Here, the camera $c$ is characterized by $f_{i}(\lambda)$, the $i$th filter response, $l(\lambda)$, the relative irradiance of the light source and $o(\lambda)$, which represents other parameters of the optical system like camera quantum efficiency or transmission of the optical elements.

2.2 Invertible Neural Networks (INNs) for physiological parameter estimation

Basic principle

INNs have been proposed recently as a new method to recover a posterior distribution $p(\mathbf{x}|\mathbf{y})$ from an observation $\mathbf{y}$ [2]. The network takes the form of a deterministic function $g$:

[TABLE]

where $\Theta$ denotes the trainable parameters and $\mathbf{z}$ are the latent variables carrying the uncertainty of the reconstruction of $\mathbf{x}$ given $\mathbf{y}$. Sampling the latent variables according to the normal distribution $\mathcal{N}(\mathbf{z};0,I)$ yields an approximation of $p(\mathbf{x}|\mathbf{y})$.

Application to physiological parameter estimation

In the context of physiological parameter estimation, we hypothesize that observing a spectrum $\mathbf{y}$ is generally not sufficient to recover the underlying tissue parameter(s) $\mathbf{x}$. Intuitively speaking, the purpose of the latent variables $\mathbf{z}$ is to capture the information necessary to recover $\mathbf{x}$ that is not already captured by $\mathbf{y}$. To recover a physiological parameter from a previously unseen spectrum $\mathbf{y}^{\text{test}}$, we repeatedly draw samples $(\mathbf{z}_{i})_{i}$ from the latent space to obtain samples ($\mathbf{y}^{\text{test}},\mathbf{z}_{i}$) that we pass through the network. The corresponding set of physiological parameters $\mathbf{x}_{i}$ yields the posterior $p(\mathbf{x}|\mathbf{y}^{test})$. Due to the invertible architecture, the network simultaneously learns (1) the forward model - i. e., how to convert tissue parameters to spectral reflectances as measured by a camera - and (2) how to recover a posterior distribution $p(\mathbf{x}|\mathbf{y})$ of tissue parameters corresponding to an observation $\mathbf{y}$.

Network architecture

The network architecture applied in this work has been adapted from [2] and can be found in Figure 3. It relies on four invertible affine coupling blocks [5], each of which is followed by a permutation layer, leading to an eight layer network in total. The purpose of the permutation layer is to improve the mixing of the different input and output channels. It adds no additional weights to the network. At initialization, a randomly chosen permutation between the input and output channels is fixed permanently. We assume that each physiological parameter has its own uncertainty associated. Hence, we choose $\dim z=\dim x=13$ in this study.

Loss functions

Four loss terms are used to train the network (cf. Figure 3):

$L_{2}$** forward loss ($L_{\text{ffit}}$):**

In the forward direction, we use an $L_{2}$ loss $L_{\text{ffit}}$ on the predicted reflectances $\mathbf{\hat{y}}$ and the true reflectances $\mathbf{y}$ to enforce good estimations for the forward process.

MMD forward loss ($L_{\text{fmmd}}$):

We apply a Maximum Mean Discrepancy (MMD) loss $L_{\text{fmmd}}$ on the latent space estimation $\mathbf{\hat{z}}$. MMD losses distinguish between distributions [8]. Here, we compare the distribution of the predicted latent variables $\mathbf{\hat{z}}$ to latent variables $\mathbf{z}$ sampled from the desired distribution $\mathcal{N}(\mathbf{z};0,I)$.

$L_{2}$** backward loss ($L_{\text{bfit}}$):**

We use the estimations $\mathbf{\hat{y}}$ and $\mathbf{\hat{z}}$ from the forward pass and perturb both quantities with additive Gaussian noise. The resulting output $\mathbf{\hat{x}}$ with zero padding $\hat{0}$ is compared to $\mathbf{x}$ with zero padding [math] via the $L_{2}$ loss $L_{\text{bfit}}$. This serves as a form of regularization, smoothing the latent space and ensuring that no critical information is hidden in low-amplitude structures in the outputs.

MMD backward loss ($L_{\text{bmmd}}$):

We compute a reverse pass through the net with reflectances $\mathbf{y}$ from the training set $T^{\text{train}}$ and latent variables $\mathbf{z}$ from $\mathcal{N}(\mathbf{z};0,I)$. The output $\mathbf{\hat{x}}$ is then passed to an MMD loss $L_{\text{bmmd}}$, which compares it to the distribution given by the training samples $\mathbf{x}$. As previous work [2] indicates that $sO_{2}$, $v_{\text{HB}}$ and $a_{\text{mie}}$ are the only tissue parameters that can potentially be recovered from multispectral measurements, we decided to only feed these slices of $\mathbf{\hat{x}}$ into $L_{\text{bmmd}}$ instead of the whole prediction.

Hyperparameter optimization

We use the training data set $T^{\text{train}}$ to perform the parameter optimization of the network and the validation data set $T^{\text{validation}}$ to prevent overfitting and for hyperparameter tuning. Particularly, we use the validation data to calibrate the width of the posterior distributions. As suggested in [20], the purpose of the calibration is that for every sample, the $\alpha$-confidence interval (0\text{\,}\%$\leq\alpha\leq$100\text{\,}\%) of the posterior contains the ground truth value in $\alpha$ of the cases. In other words, for each value of $\alpha$ exactly a fraction $\alpha$ of the ground truth values shall be inliers of the corresponding $\alpha$-confidence interval. We optimize the parameters using the validation set $T^{\text{valid}}$ to enforce this behavior as best as possible.

2.3 Performance assessment

We quantify the uncertainty of an inference based on two key parameters: The presence of multiple modes and the width of the posterior.

Given samples following the posterior $p(\mathbf{x}|\mathbf{y})$ our approach to automatic mode detection relies on computing a kernel density estimation for $p(\mathbf{x}|\mathbf{y})$ which has the advantage of being easily sampled. This then allows us to compute the corresponding relative maxima of $p(\mathbf{x}|\mathbf{y})$. A posterior is classified as multi-modal, if its standard deviation is less than half of the prior’s standard deviation, our algorithm finds more than one relative maximum, and these maxima are further than a certain threshold apart ($sO_{2}$: $3\text{\,}\mathrm{p}\mathrm{p}$, $v_{\text{HB}}$: $0.3\text{\,}\mathrm{p}\mathrm{p}$). Furthermore, maxima whose intensity is less than 80% of the main (i. e. highest) maximum are ignored. All remaining posteriors are classified as uni-modal.

To assess the performance of a camera, the INN is applied to $T^{\text{test}}$, and the automatic mode detection (Section 2.3) is run on each posterior $p(\mathbf{x}|\mathbf{y}^{\text{test}})$. Next, the following metrics are computed:

• •

Percentage of multiple modes (MM): The percentage of multi-modal posterior distributions. We do this as a means to judge how well-posed the inversion is for the different cameras.

• •

Root-mean-square error (RMSE): We utilize maximum a posteriori probability (MAP) estimate as a predictor for the physiological parameters and give the root-mean-square error of these estimations against the ground truth.

• •

68% confidence interval width (W): We report the median interval width of the 68% confidence interval as a measure of the width of the posterior distributions.

3 Experiments and Results

The purpose of the experiments was to confirm the realism of our simulation pipeline (Section 3.1) and to apply our setup to the task of comparative camera assessment (Section 3.2).

3.1 Realism of Simulation Pipeline

The simulation pipeline applied for comparative camera assessment features two potential sources of error: (1) Errors in the conversion of the simulated high resolution spectrum to multispectral measurements (sec. 3.1.1) and (2) wrong model assumptions in the generic tissue model and hence errors in the simulated spectra (sec. 3.1.2). We will address both issues in this order in the following paragraphs.

3.1.1 Virtual Camera

The realism of the simulated data relies crucially on the validity of our virtual cameras. In order to explore this, we measured color tiles (X-Rite ColorChecker®classic, Grand Rapids, MI, USA) which have a well defined spectrum using a HR2000+ spectrometer (Ocean Optics, Largo, FL, USA) and a Pixelteq SpectroCam™, which is a 8 band multispectral camera. Using the filter response functions of the SpectroCam, we transformed the high resolution spectrum to a virtual SpectroCam spectrum. To perform this experiment we used three color tiles (blue, green and red) and averaged five SpectroCam measurements. The measured intensities were normalized. As shown in Figure 4, the simulated data is in very close agreement with the real measurements.

3.1.2 Tissue Model

Due to the lack of a reliable gold standard method for measuring optical tissue properties in vivo, validation of the tissue model is not straightforward. Previous work has addressed this issue by comparing real measurements of tissue with simulated spectra [30]. If the accuracy of the virtual camera used can be assumed to be acceptable, deviations between real and simulated data can primarily be attributed to differences in the tissue composition. The tissue model applied in this study has been validated in a previous publication [30] using multispectral data from several different porcine abdominal organs. To confirm these findings we additionally acquired measurements from a porcine brain and a human kidney. The brain was measured using the same SpectroCam as in Section 3.1.1. For the human kidney, we used a 16-band camera. We performed a principal component analysis (PCA) on the data generated by our tissue model (adapted to the appropriate camera) and a kernel density estimation (KDE) on the first two principal components. Afterwards, we projected the measured data on those same components. The result can be found in Figure 5. Clearly, all the organ data points lie within the distribution of the simulated data of our tissue model.

3.2 Comparative Camera Assessment

The main purpose of our experiments was to evaluate our assessment framework in the specific context of multispectral camera selection for physiological parameters estimation.

3.2.1 Experimental Data

We applied the Monte Carlo-based method described in Section 2.1 to generate 20,000 data points representing spectral reflectances using the tissue parameter distributions summarized in Table 1. If we give a range for a parameter this parameter is sampled uniformly from this range. If we give a value with standard deviation this parameter is sampled according to a normal distribution with this expectation value and standard deviation. For each camera setup investigated here, these reflectances were converted into (simulated) camera measurements considering the optical properties of the setup. For each setup, we reserved 70% of the data for training ($T^{\text{train}}$), 5% for hyperparameter tuning ($T^{\text{validation}}$) and 25% for performance assessment ($T^{\text{test}}$). To test our assessment framework, we assessed three camera designs that have been applied in previous work [12, 19, 29] in a comparative manner. To obtain a lower bound on the achievable uncertainty, we complemented these realistic cameras by a virtual camera with nearly optimal design. The cameras are characterized by the following filter responses $f_{i}$ (cf.  Figure 6):

3-med**:**

3-band camera optimized for medical imaging use, as described in [12].

3-nRGB**:**

3-band camera whose bands’ centers coincide with the standard RGB bands, as described in [19].

8-med**:**

Pixelteq SpectroCam, a 8-band-camera optimized for medical imaging use, as described in [29]. This is the same camera as used in the experiments in Section 3.1.

27-equi**:**

As a close to optimal camera, we used a camera with a filter response featuring a (unrealistically) narrow band every $10\text{\,}\mathrm{nm}$. As our experimental data is based on presimulated data in the range of $450\text{\,}\mathrm{nm}$ to $720\text{\,}\mathrm{nm}$, this leads to a ‘27 band camera’.

The remaining parameters $l(\lambda)$ and $o(\lambda)$ were set to 1 for all cameras.

3.2.2 Results

Figure 7 provides representative examples for the posteriors generated by our INN. The calibration errors for the four different cameras are presented in Figure 8 for the physiological parameters tissue oxygenation ($sO_{2}$) and blood volume fraction ($v_{\text{HB}}$). We see that the calibration curves closely follow the identity. In the case of $v_{\text{HB}}$, the 3-med camera is ‘underconfident’ for larger values which would make estimations based on the confidence intervals in this range less reliable.

Table 2 shows the performance of the four different cameras using the metrics presented in Section 2.3. All computations were performed on the test set $T^{\text{test}}$. As expected, the scores generally improve with an increasing number of spectral bands. An interesting observation is that the 3-band camera designed for medical use (3-med) has a higher RMSE compared to the camera whose design was inspired by standard RGB cameras (3-nRGB), yet, it features a substantially reduced number of multiple mode posteriors (3.6% vs 9.3%).

For all cameras except the 3-nRGB, there are only few multiple mode posteriors for $sO_{2}$ reconstruction. Figure 9(a) shows the estimations of the 27-equi and the 3-med camera which show generally good reconstructive performance and the possibility of outlier detection via the width of the posteriors.

In contrast, our results suggest that $v_{\text{HB}}$ cannot be recovered from any of the cameras with high certainty. In fact, the percentage of samples with $W\geq 0.5\cdot\sigma_{prior}$, (where $\sigma_{prior}$ represents the standard deviation of the corresponding the prior) is greater than 50% for all four cameras. The poor performance is illustrated in Figure 9(b). We see that the 27-equi camera still performs better than the 3-med camera, but none of them show good performance for high $v_{\text{HB}}$ values. This general trend is also true for the other two cameras. Note that since most posteriors were even wider than the priors, they did not qualify as a candidate for the multiple mode mode detection algorithm (cf. Section 2.3) explaining the low MM.

Furthermore, although W seems reasonable in absolute terms, comparing it to twice the standard deviation of the prior distribution reveals that the median width of the posteriors goes as high as 93% for the 3-med camera, indicating that $v_{\text{HB}}$ is effectively unrecoverable. For $sO_{2}$ the values range from 4% in the 27-equi case to 15% in the 4-med case.

4 Discussion

Meaningful performance assessment and benchmarking are crucial for advancing research and practice. Several publications, however (cf. e. g. [18]), suggest that the metrics chosen are not always well-suited for a specific assessment goal. In the context of multispectral intra-operative imaging, for example, camera assessment has typically been restricted to determining descriptive statistics on error metrics that quantify the difference between the estimations of an algorithm and reference (gold standard) results [25, 29, 30]. An advantage of this approach is that the error metrics are straightforward to compute and interpret. On the other hand, such performance measures suffer from the fact that they do not reveal important insights with respect to why methods perform poorly. In particular, they do not account for the different types of uncertainties that may occur when recovering tissue parameters from camera measurements. An interesting practical example is the 3-band camera designed for medical use [12] and investigated here. While it features a higher RMSE compared to a 3-band camera based on the standard RGB design, recovery of tissue parameters is substantially less ambiguous, as indicated by the reduced number of multiple modes.

To address the issues related to commonly applied approaches to camera design, selection and performance assessment, we present a novel approach to camera assessment which provides the following key advantages compared to previously proposed methods:

Extended scope: The topic of camera design is closely linked to that of band selection [28]. To our knowledge, however, none of the approaches proposed in this field addresses the potential inherent ambiguity associated with the recovery of physiological parameters. To overcome this bottleneck, we propose moving beyond point estimates and mapping measurements to a full posterior probability distribution. Analysis of the posteriors not only provides us with a means for quantifying the uncertainty related to a specific measurement but also allows for a fundamental theoretical analysis about which tissue properties can in principle be recovered with the present camera. 2. 2.

No need for acquisition of real data: Many approaches to band/camera selection rely on acquisition of real data [9, 10, 21, 22, 27]. Yet, acquisition of real data for a given application is often impractical due to budget constraints (no money to purchase a whole range of cameras) or ethical issues. We address this issue by performing the comparative assessment in silico. Experiments with a whole range of porcine and human organs confirm the realism of our simulation framework.

The above computations show that these networks can compute the same error metrics as before (e. g. RMSE) while having the potential for finer differentiation through additional metrics (e. g. number of modes or width of posterior).

While it is straightforward to compute the widths of the posteriors, fully-automatic multiple mode detection is not trivial due to the many parameters involved. For example, the posteriors are only implicitly given by a number of samples generated according to a latent space sample. This fact alone introduces statistical fluctuations into the estimated posterior. A kernel density estimation can smoothen out these effects, but at the cost of introducing a bandwidth parameter with a high impact on the number of resolved maxima. In addition, outliers must be handled in order to avoid faulty signals at the boundary of the posterior.

The calibration of our models suggest that while the confidence of our posteriors is already good, there is still room for improvement. In particular, the calibration of $v_{\text{HB}}$ for the 3-med camera is off for larger confidences. In future studies which aim at finer differentiation between the observed cameras this would have to be remedied. We are confident that this can be achieved keeping in mind the convincing results for the other three cameras.

Another obstacle which learned methods have to sidestep are the so called out of distribution samples. The performance of our algorithm can only be guaranteed on data that is similar to the training data. In general, this problem is difficult to tackle. In our case, the PCA projections of the organ measurements show exemplary that the spectra of many interesting objects, like internal organs, are in fact in our training distribution.

The color tile experiments together with the measured organ spectra suggest the validity of our simulation framework. A natural next step would be to test the performance of our method on real data. To achieve this, there remains a key challenge: real data will always be subject to noise which needs to be handled adequately by our algorithm. One approach would be to average the spectra either by using a higher integration time or by averaging multiple measurements. However, if there are time constraints, for example induced by organ movement, there are limits to the amount of averaging possible. Another approach would be to incorporate a realistic noise model in the simulation framework to account for it during training. This would circumvent the time constraint as the evaluation time of the network trained with this new data set would not change.

Another interesting direction for future work is to apply the framework to additional cameras that are widely used in a clinical context (e. g. RGB or narrow band cameras). We expect some obstacles with regard to the extension to 2-band cameras as there is just very little information left for a multi parameter reconstruction. Additionally, these cameras would need a larger range of simulated wavelengths compared to the data set that we based our work on [29]. However, extending the framework to these ranges should be straight forward.

Additionally, while this study focused on intra-operative optical imaging, the concept of performance assessment using INNs could easily be transferred to other fields of research. Clearly, any imaging modality with pixelwise spectral information is a prime candidate. For larger image context, the INNs are still in active development. Because of their peculiar structure, the hidden layer size is the same as the input and output dimension leading to very large networks when images are to be processed as a whole. One example of an imaging modality where it might be fruitful to apply our INN method is the field of quantitative photoacoustic imaging (qPAI). It has been shown before that qPAI is an ill-posed inverse problem in theory [4]. However, to the best of our knowledge an in silico or even in vivo analysis of the practical implications of this non-uniqueness has not been conducted. The ability to detect ambiguous reconstructions of physiological parameters seems like a promising candidate to close this gap.

In conclusion, we have presented a novel method for performance assessment of optical cameras bearing the potential to measure the well-posedness of the inverse problem. Future work should focus on the evaluation steps necessary to fully harness the power of the computed posterior distributions. In particular, robust mode detection algorithms seem like a fruitful area for further investigation in order to quantify the uniqueness of the reconstruction.