A deep learning framework for quality assessment and restoration in video endoscopy

TL;DR

This paper presents a comprehensive deep learning framework for detecting, classifying, assessing quality, and restoring artifacts in endoscopy videos, improving accuracy and efficiency for clinical applications.

Contribution

It introduces a fully automatic system that detects multiple artifacts, assesses frame quality, and restores corrupted frames using advanced neural networks, surpassing prior methods.

Findings

Detector achieves highest mean average precision of 49.0

Restoration models significantly outperform previous methods

Preserves 68.7% of frames, 25% more than raw videos

Abstract

Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. Artifacts such as motion blur, bubbles, specular reflections, floating objects and pixel saturation impede the visual interpretation and the automated analysis of endoscopy videos. Given the widespread use of endoscopy in different clinical applications, we contend that the robust and reliable identification of such artifacts and the automated restoration of corrupted video frames is a fundamental medical imaging problem. Existing state-of-the-art methods only deal with the detection and restoration of selected artifacts. However, typically endoscopy videos contain numerous artifacts which motivates to establish a comprehensive solution. We propose a fully automatic framework that can: 1) detect and classify six different primary artifacts, 2) provide a quality score for each frame and 3) restore mildly corrupted frames. To detect different artifacts our framework exploits fast multi-scale, single stage convolutional neural network detector. We introduce a quality metric to assess frame quality and predict image restoration success. Generative adversarial networks with carefully chosen regularization are finally used to restore corrupted frames. Our detector yields the highest mean average precision (mAP at 5% threshold) of 49.0 and the lowest computational time of 88 ms allowing for accurate real-time processing. Our restoration models for blind deblurring, saturation correction and inpainting demonstrate significant improvements over previous methods. On a set of 10 test videos we show that our approach preserves an average of 68.7% which is 25% more frames than that retained from the raw videos.

Figures

Tables

Table 1. Table 1: Computational models used for individual artifact classes.

artifact type	Restoration method
Motion blur	CGAN + $l2$-contextual + high-frequency losses
specularity/bubbles/misc. artifacts	CGAN + $l1$-contextual loss
saturation	CGAN + $l2$-contextual loss + CRT transform
Low contrast	same as saturation (reversed training set)

Table 2. Table 2: Artifact detection results on test set with different neural network architectures. All timings are reported on a single 6GB NVIDIA GTX Titan Black GPU and is the average time for a single 512x512 image (possibly rescaled on input as indicated) evaluated over all 129 test images. Total number of ground truth boxes = 644 boxes.

Method	Backbone	Input Size	mAP5	mAP25	mAP50	IoU25	Predict Boxes	Time (ms)
Faster R-CNN[34]	Resnet50	600x600	44.9	40.4	29.5	28.3	835	555222Python Keras 2.0, (Tensorflow 1.2 backend) Code.
RetinaNet[23]	Resnet50	608x608	43.8	41.2	34.7	38.9	576	103333PyTorch 0.4 Code.
YOLOv3[33]	darknet53	512x512	47.4	44.3	35.1	24.2	1252	95444Python call of Darknet trained network.
YOLOv3	darknet53	608x608	48.1	45.2	33.2	21.4	1300	1164
YOLOv3-spp	darknet53	512x512	49.0	45.7	34.7	24.4	1120	88

Table 3. Table 3: Class-specific average precision (AP) of the different object detection networks.

Method	Spec.	Sat.	Arte.	Blur	Cont.	Bubb.
Faster R-CNN[34]	20.7	71.0	35.1	14.5	58.7	42.4
RetinaNet[23]	33.1	42.9	39.8	7.2	73.6	50.6
YOLOv3[33]	40.0	50.4	44.3	11.6	70.8	48.9
YOLOv3-spp	34.7	55.7	48.0	7.5	72.1	55.9

Table 4. Table 4: Peak signal-to-noise ratio (PSNR) and the structural similarity measure (SSIM) for randomly selected images with different motion blur.

Method	Metric	Images with varying motion blur
		#80	#99	#102	#113	#116
CGAN$+$cont. &	PSNR	25.22	28.14	27.28	23.41	24.81
HF feature loss	SSIM	0.998	0.997	0.993	0.980	0.992
deblur	PSNR	25.17	27.93	26.96	23.40	24.81
GAN [21]	SSIM	0.998	0.997	0.992	0.979	0.992
SRN	PSNR	24.61	27.50	25.02	22.23	22.00
DeblurNet [39]	SSIM	0.995	0.996	0.990	0.970	0.970
TV-deconv	PSNR	24.25	26.72	24.75	21.69	22.20
[11]	SSIM	0.966	0.994	0.988	0.966	0.983

Table 5. Table 5: Average PSNR ( P ​ S ​ N ​ R ¯ ¯ 𝑃 𝑆 𝑁 𝑅 \overline{PSNR} ) and average SSIM ( S ​ S ​ I ​ M ¯ ¯ 𝑆 𝑆 𝐼 𝑀 \overline{SSIM} ) for image sequences in test trajectories both with added high-frequency (HF) feature loss (proposed) and only contextual loss  [ 21 ] in conditional GAN model.

Method	Metric	Image sequences
		#1	#2	#3
CGAN$+$cont. &	$\overline{PSNR}$	25.80	24.65	21.25
HF feature-loss	$\overline{SSIM}$	0.997	0.980	0.970
deblur	$\overline{PSNR}$	25.68	24.37	21.08
CGAN [21]	$\overline{SSIM}$	0.996	0.977	0.968

Table 6. Table 6: Average PSNR ( P ​ S ​ N ​ R ¯ ¯ 𝑃 𝑆 𝑁 𝑅 \overline{PSNR} ) and average SSIM ( S ​ S ​ I ​ M ¯ ¯ 𝑆 𝑆 𝐼 𝑀 \overline{SSIM} ) for 19 randomly selected saturated images in our simulated data set using CycleGAN. Quality assessment (QA) for simulated images, l ​ 2 𝑙 2 l2 -contexual CGAN, and post-processing using color retransfer (CRT) method are provided.

Metric	QA for different stages
	CyleGAN	$l2$-contexual	post-process
	simulation	CGAN	CRT
$\overline{PSNR}$	27.892	28.622	28.335
$\overline{SSIM}$	0.905	0.964	0.944
$\overline{VIF}$	0.808	0.810	0.818
$\overline{RECO}$	1.091	1.313	1.512

Table 7. Table 7: Average values for PSNR, VIF  [ 36 ] , and RECO  [ 5 ] ) metrics for restoration of missing pixels for masks covering 5% and 12% of total image pixels ( 512 × 512 512 512 512\times 512 pixels) with 21 randomly sampled rectangular boxes on 20 randomly selected images from 3 different patient videos.

Method	5% of total pixels			12% of total pixels			$\overline{t}$
	PSNR	VIF	RECO	PSNR	VIF	RECO	s
TV-based [11]	45.130	0.947	0.984	40.970	0.881	0.975	392.0
Patch-based [30]	43.440	0.940	0.990	39.520	0.871	0.984	35.0
$l1$-cont. CGAN	43.487	0.950	0.992	39.693	0.883	0.983	2.5

Equations

$$QS=\lfloor 1-\sum _{\mathcal{B}}({\lambda }_A{W}_C{W}_A+{\lambda }_L{W}_C{W}_L){\rfloor }_0,$$

$$I(t)=F\left[(h\ast f(t)+\eta {)}^{\gamma }\right],$$

$$\underset{G}{\min }\underset{D}{\max }V(D,G)={\mathbb{E}}_{x\sim p_{data}(x)}[\log D(x)]+$$

$${\mathbb{E}}_{z\sim p_{noise}(z)}[\log (1-D(G(z)))]$$

$$\underset{G}{\min }\underset{D}{\max }{V}_{cond}(D,G)={\mathbb{E}}_{x,y\sim p_{data}(x,y)}[\log D(x|y)]+$$

$${\mathbb{E}}_{y\sim p_y,z\sim p_z}[\log (1-D(G(z|y),y))]$$

$$\underset{G}{\min }\underset{D}{\max }{V}_{cond}^{{}^{\prime }}(D,G)=V_{cond}+\sum _i{\lambda \parallel x_{rea{l}_i}-G(z_i|y_i)\parallel }_l,$$

$$I_t^{{}^{\prime }}={\mathbf{\Sigma }}_s^{1/2}{\mathbf{\Sigma }}_t^{-1/2}(I_t-{\mu }_t)+{\mu }_s,$$

Full text

We propose a fully automatic framework that can: 1) detect and classify six different primary artifacts, 2) provide a quality score for each frame and 3) restore mildly corrupted frames. To detect different artifacts our framework exploits fast multi-scale, single stage convolutional neural network detector. We introduce a quality metric to assess frame quality and predict image restoration success. Generative adversarial networks with carefully chosen regularization are finally used to restore corrupted frames.

Our detector yields the highest mean average precision (mAP at 5% threshold) of 49.0 and the lowest computational time of 88 ms allowing for accurate real-time processing. Our restoration models for blind deblurring, saturation correction and inpainting demonstrate significant improvements over previous methods. On a set of 10 test videos we show that our approach preserves an average of 68.7% which is 25% more frames than that retained from the raw videos.

1 Introduction

Originally used to image the esophagus, stomach and colon, miniaturization of hardware and improvement of imaging sensors now enable endoscopy of the ear, nose, throat, heart, urinary tract, joints, and abdomen. Common to these endoscopy applications, presence of different imaging artifacts pose significant challenges in monitoring disease progression. The camera in the endoscope is embedded in a long flexible tube. Any small hand motion can cause severe motion artifacts in recorded videos. The light, required for illumination, can interact with tissue and surrounding fluid generating very bright pixel areas (either due to specularity or pixel saturation). Different viewing angles and occlusions can result in contrast issues due to underexposure. Additionally, similar to any other complex real-world imaging applications, visual clutters of debris, liquid, bubbles, etc., can limit the visual understanding of the underlying tissue. In this study, we are thus considering the following artifacts: specular reflections, pixel saturation, motion blur, contrast and undesired visual clutter. Not only do such artifacts occlude the tissue/organ of interest during diagnosis and treatment, they also adversely affect any computer assisted endoscopy methods (e.g., video mosaicking for follow-ups and archiving, video-frame retrieval for reporting etc.).

Chikkerur et al. [8] and Menor and colleagues [26] have studied video frame quality assessment methods. While they introduce and review very useful global video quality metrics; neither information regarding the cause of frame quality degradation nor the degraded regions could be identified for frame restoration. In general, utilizing these quality scores [8]-​​[26] only allows for the removal of frames corrupted with artifacts without considering the severity of each artifact type. Such simple removal of corrupted frames can severely reduce the information content of videos and affect their overall temporal smoothness. One adverse effect of this, for example, can be on mosaicking methods that require at least $60\%$ overlap in successive temporal frames to succeed [2]. Artifacts are thus the primary obstacles in developing effective and reliable computer assisted endoscopy tools. The precise identification, classification and -if possible- restoration are critical to perform a downstream analysis of the video data.

Detecting multiple artifacts and providing adequate restoration is highly challenging. To date, most research groups have studied only specific artifacts in endoscopic imaging [25, 38, 40, 1]. For example, deblurring of wireless capsule endoscopy images utilizing a total variational (TV) approach was proposed in [25]. TV-based de-blurring is however parameter sensitive and requires geometrical features to perform well. Endoscopic images have very sparse features and lack geometrically prominent structures. Both hand-crafted features [38, 40, 1, 28] and neural networks [17] have been used to restore specular reflections. A major drawback of these existing restoration techniques is that heuristically chosen image intensities are compared with neighboring (local) image pixels. In general, both local and global information is required for realistic frame restoration. One common limitation of almost all the methods is that they only address one particular artifact class, while naturally various different effects corrupt endoscopy videos. For example, both ‘specularities’ and a water ‘bubble’ can be present in the same frame. Endoscopists also dynamically switch between different modalities during acquisition (e.g., normal brightfield (BF), acetic acid, narrow band imaging (NBI) or fluorescence light (FL)) to better highlight specific pathological features. Finally, inter-patient variation is significant even when viewed under the same modality. Existing methods fail to adequately address all of these challenges. In addition to addressing one type of imaging artifact, only one imaging modality and a single patient video sequence are considered in most of endoscopy-based image analysis literature [38, 40, 1, 17, 28]. The use of small size data sets in these studies also raises concern regarding method generalization to image variabilities often present in endoscopic data. For example, in [1] only 100 randomly selected images were used to train the Support Vector Machine (SVM) for detecting specular regions.

In this paper, we propose a systematic and comprehensive approach to the problem. Our framework addresses the precise detection and localisation of six different artifacts and introduces artifact type specific restoration of mildly affected frames. Unlike previous methods [25, 38, 40, 1, 17, 28] that require manual adjustment of parameter settings or the use of hand-crafted features only suitable for specific artifacts, we propose to use multiple class artifact detection and restoration methods utilizing multi-patient and multi-modal video frames. Such an approach decreases false classification rate and better generalizes both detection and frame restoration methods. Reliable multi-class detection is made possible through a multi-scale and deep convolutional neural network based object detection which can efficiently generalize multi-class artifact detection in cross patients and cross modality present in endoscopic data. Realistic frame restoration is achieved using Generative adversarial networks (GANs, [13]). While our work is built on these approaches, substantial additional work has been necessary to avoid the introduction of additional artifacts and disruptions to the the overall visual coherence. In order to achieve this we introduce artifact type dependent regularization. A novel edge-based regularization and restoration is proposed for de-blurring. Restoration of large saturated pixel areas using GAN is of its first kind and have never been addressed in literature. In order to handle the color shift we introduce a novel color-transfer technique in this scheme. For tackling with artifacts like debris, bubbles, and other misc. artifacts we apply a complete restoration of pixels based on global contextual regularization scheme. Additionally, we condition each of GAN model with prior image information. We demonstrate that such carefully chosen models can lead to both high-quality and very realistic frame restoration.

To our knowledge, this is the first attempt to propose a systematic and general approach that handles cross modality, and inter-patient video data for both automatic detection of multiple artifacts present in endoscopic data and their subsequent restorations. We use 7 unique patient videos (gastroesophageal, selected from large cohort of 200 videos) for training and 10 different videos for extensive validations. Our experiments utilizing well-established video quality assessment metrics illustrate the effectiveness of our approaches. In addition, quality of the restored frames has also been evaluated by two experienced endoscopists. A score based on visual improvement, importance, and presence or absence of any artificially introduced artifact in our restored frames were provided by these experts.

The remainder of this article is organized as follows. In Section 2, we introduce our endoscopy data set for artifact detection. Section 3 details our proposed approaches for artifact detection and endoscopic video frame restoration. In this section we also review closely related works associated with each process. In Section 4, we present experiments and results for each step of our framework to show the efficacy of individual methods. Finally, in Section 5 we conclude the paper and outline directions for future work.

2 Material

Our artifact detection data set consists of a total of 1290 endoscopy images (resized to 512 x 512 pixels) from two operating modalities; normal bright field (BF), and narrow-band imaging (NBI) sampled from 7 unique patient videos selected from a cohort of 200 endoscopic videos for training data. The selection was based on number of representative artifacts present in these videos and texture variability of the underlying esophagus. Two experts annotated a total of 6504 artifacts using bounding boxes where each annotation is classified as:

blur - streaking from fast camera motion 2. 2.

bubbles - water bubbles that distorts appearance of the underlying tissue 3. 3.

specularity - mirror-like surface reflection 4. 4.

saturation - overexposed bright pixel areas 5. 5.

contrast - low contrast areas from underexposure or occlusion 6. 6.

misc. artifact (also referred as ‘artifact‘ in this paper)- miscellaneous artifacts; e.g., chromatic aberration, debris, imaging artifacts etc.

A 90%-10% split was used to construct the train-test set for object detection resulting in 1161 and 129 images and 5860 and 644 bounding boxes, respectively. In general, the training and testing data exhibits the same class distribution (see Fig.1 (top row)) and similar bounding boxes (roughly square) but either small with average widths less than 0.2 or large with widths greater than 0.5 (see Fig.1 (bottom row)). Multiple annotations are used in case a given region contains multiple artifacts.

3 Method

3.1 Overall approach

The step-by-step procedure for automatic detection of multiple artifacts and frame restoration of endoscopic videos is presented in Fig. 2. It is to be noted that a single frame can be corrupted by multiple artifacts and each artifact class can affect endoscopic frames differently. Therefore, their restoration process is very likely to affect the final restoration result.

Multiple instance object detection is used to discriminate between the six different types of artifacts (see Section 2) and normal appearance. For each frame a quality score (QS, refer Section 3.3) is computed based on the type, area and location of the identified artifacts to reflect the feasibility of complete image restoration via the sequential restoration process depicted in Fig. 2. The scaling of our QS score is set such that we differentiate between severely corrupted frames ($QS<0.5$), mildly corrupted frames ($0.5\leq OS\leq 0.95$), and frames of high quality ($QS>0.95$). Severely corrupted frames are discarded without any further processing. The proposed image restoration methods are applied to mildly corrupted frames only. In order to guarantee a faithful restoration, mildly corrupted frames go through our proposed sequential framework. All remaining frames are directly concatenated into the final list without any processing.

3.2 Artifact region detection

Recent research in computer vision provides us with object detectors that are both robust and suitable for real-time applications. Here, we propose to use a multi-scale deep object detection model for identifying the different artifacts in real-time. Even by itself, real-time artifact detection is already of practical value. For example, the detection results can be used to provide endoscopists with feedback during data acquisition. After detection, additional post-processing using traditional image processing methods can be used to determine the precise boundary of a corrupted region in the image.

Today, deep learning enables us to construct object detectors that generalise traditional hand-crafted ‘sliding-window’ object classification approaches (e.g., Viola-Jones [43]). Earlier attempts of including OverFeat [35] and R-CNN [12] demonstrated the power of convolutional neural networks (CNNs) to learn relevant features and detect objects using a fixed number of pre-generated candidate object region proposals [42]. Faster R-CNNs [34] first introduced a fully trainable end-to-end network yielding an initial region proposal network and successive classifications of the proposed regions without intermediate processing. Since region proposal generation precedes bounding box detection sequentially, this architecture is known as a two-stage detector. Though very accurate, a primary drawback is its slow inference and extensive training. You Only Look Once (YOLO, [32]) simplified Faster R-CNNs to predict simultaneously class and bounding box coordinates using a single CNN and a single loss function with good performance and significantly faster inference time. This simultaneous detection is known as a one-stage detector. Compared to two-stage detectors, single-stage detectors mainly suffer two issues: high false detection due to 1) presence of varied size objects and 2) high initial number of anchor boxes requirement that necessitates more accurate positive box mining. The former is corrected by predicting bounding boxes at multiple scales using feature pyramids [14]-​​[22]. To address the latter, RetinaNet [23] introduced a new focal loss which adjusts the propagated loss to focus more on hard, misclassified samples. Recently, YOLOv3 [33] simplified the RetinaNet architecture with further speed improvements. Bounding boxes are predicted only at 3 different scales (unlike 5 in RetinaNet) utilizing objectness score and an independent logistic regression to enable the detection of objects belonging to multiple classes unlike focal loss in RetinaNet. Collectively, Faster R-CNN, RetinaNet and YOLOv3 define the current state-of-the-art detection envelope of accuracy vs speed on the popular natural images benchmark COCO data set [24].

We investigated the Faster R-CNN, RetinaNet and YOLOv3 architectures for artifact detection. Validated open source codes are available for all of these architectures. Experimentally, we chose to incorporate YOLOv3 with spatial pyramid pooling (YOLOv3-spp) for robust detection and improved inference time for endoscopic artifacts detection. Spatial pyramid pooling allowed to pool features from sub-image regions utilizing computed single-stage CNN features at multiple-scales from YOLOv3 architecture. In addition to the boost in the inference speed, incorporating spatial pyramid pooling decreased false positive detections compared to classical YOLOv3 method (see Section 4.2). YOLOv3-spp provided an excellent feature for accuracy-speed trade-off which are the main requirements for usage in clinical settings. Examples of the detected boxes using YOLOv3-spp are shown in Fig. 3.

3.3 Quality score

Quality assessment is important in video endoscopy as image corruption largely affects image analysis methods. However, it is likely that not all frames are corrupted in same proportion. Depending on the amount and type of artifact present in frames realistic frame restoration can be possible. However, such frame grading needs to be carefully determined. Here, we propose a frame quality score (QS) based on: a) type, b) area and c) location of the detected artifacts. Weights are assigned to each of these categories and a mean weight is computed as the quality score. Weights are assigned to each type based on the ease of restoration, e.g., an entire blurred image can still be restored but the same would not apply with misc. artifacts. Thus, misc. artifacts are assigned a higher weight than blur. Similarly, the area and location of detected artifacts in each frame are important. A centrally located imaging artifact with large area detrimentally degrades image information beyond restoration. Below we describe our weighting scheme:

• •

Class weight ($W_{C}$): misc. artifact (0.50), specularity (0.20), saturation (0.10), blur (0.05), contrast (0.05), bubbles (0.10)

• •

Area weight ($W_{A}$): percentage of the total image area occupied by all detected artifact areas and normal areas

• •

Location weight ($W_{L}$): center (0.5), left (0.25), right (0.25), top (0.25), bottom (0.25), top-left (0.125), top-right (0.125), bottom-left (0.125), bottom-right (0.125).

The final QS is computed as:

[TABLE]

where $\mathcal{B}$ denotes the set of bounding boxes associated to each detected artifact, $\lambda_{A}$, $\lambda_{L}$ are constants that weight the relative contributions of area and location. We have used $\lambda_{{A}}=0.5,\lambda_{{L}}=0.5$ in our experiments. However, for frames with few detected artifacts (less than 5) such weighting scheme underscores (especially if large area artifacts are present) thus $\lambda_{{A}}=1,\lambda_{{L}}=1$ is used for these cases. Note that QS score in Eq. (1) is lower-bounded by 0.

A fixed threshold is user-specified to determine the frames kept for image restoration. Examples of the proposed quality score applied to real data are shown in Fig. 4. The video frame in Fig. 4 (left) has mostly a contrast problem (i.e., low $W_{C}$) so despite its central location (see blue box) and large area the frame intensity can be restored ($\therefore$QS$=$0.75). However Fig. 4 (right) has many misc. artifacts (high $W_{C}$) and specular areas located centrally centrally (c.f. green, red boxes, $\therefore$ QS=0.23) which inhibits realistic frame restoration so the frame is discarded.

3.4 Image restoration

Formulating the reconstruction of the true signal given the noisy and corrupted input image $I$ as an optimization or estimation problem demands a well-motivated mathematical model. Unfortunately, the various different types of artifacts induce a level of complexity that make this endeavor very challenging. Assuming image noise to be additive and approximating motion blur as a linear convolution with an unknown kernel is reasonable and in line with previous attempts to the problem. In addition, contrast and pixel saturation problems can be formulated as a non-linear gamma correction. Other remaining artifacts (e.g., specularities, bubbles and imaging artifacts) which are due to combined processes of these phenomena can be assumed as a function of the entire process. The corrupted noisy video frame can thus be approximated as:

[TABLE]

where $\eta$ denotes the additive noise induced by the imaging system, the convolution with $h$ the approximation to the induced motion blur, $\gamma$ captures the over- and under-exposed regions and $F$ is a generalized non-linear function that models capturing other artifacts as well (including specularities, bubbles and imaging artifacts) or a combination of them. This model motivates why the restoration of the video frames is structured into separate processing steps, which are implemented as deep learning models.

Image restoration is the process of generating realistic and noise free image pixels from corrupted image pixels. In endoscopic frame restoration, depending upon the artifact type, the goal is either the generation of an entire noise-free image or pixel inpainting of undesirable pixels using surrounding pixel information [4]. For multi-class endoscopic artifact restoration, we require 1) frame deblurring when $h(.)$ is unknown, i.e. a blind deblurring task, 2) minimize the effect of contract imbalance (correction for over- and under-exposed regions) in frames, i.e. $\gamma$ correction, 3) replace specular pixels and those with imaging artifacts or debris with inpainting, i.e. correction for additive noise $\eta(.)$ or a combined non-linear function $F(.)$. Due to the higher likelihood of the presence of multiple artifacts in a single frame, unordered restoration of these artifacts can further annihilate frame quality. We therefore propose an adaptive sequential restoration process that account for the nature of individual artifact types (see Fig. 2).

Recently, GANs [13] have been successfully applied to image-to-image translation problems using limited training data. Here, a generator $G$ ‘generates’ a sample $G(z)$ from a random noise distribution ($p_{noise}(z)$ with $z\!\!\sim\!\!\mathcal{N}(0,\sigma^{2}\textit{I})$) while a separate discriminator network tries to distinguish between the real target images ($p_{data}(x)$ with assumed $x\!\sim\!$ non-zero mean Gaussian) and the fake image generated by the generator. The objective function $V$ is therefore a min-max problem in this case:

[TABLE]

In practice, the generator model in Eq. (3.4) is highly non-convex, unstable and slow to train as samples are generated from random input noise. Various groups [27, 47, 18, 19] have provided ways to address this problem and achieved improvements in reconstruction quality and numerical stability as well as a reduction in computation time. One popular way to ensure the stability of the generator output is by conditioning the GAN on prior information (e.g., the class label ‘$y$‘ in CGAN, [27]). The objective function $V_{cond}$ for CGAN can be written as:

[TABLE]

Another efficient method is regularizing the generator using contextual losses (e.g., pix2pix [18], deblurGAN [21]). In [9] regularizing the discriminator and generator significantly helped to improve visual quality. We train such conditional generative adversarial models [27] (CGAN) embedding artifact class dependent contextual losses, see Table 1 for effective restoration. When restoring frames, the detected artifact types are used to decide which CGAN model is applied and in what order (also see Fig. 2).

3.4.1 Motion blur

Motion blur is a common problem in endoscopy videos. Unlike static images, motion blur is often non-uniform with unknown kernels $h(.)$ (see Eq. (2)) in video frame data. Several blind-deconvolution have been applied to motion deblurring. These range from classical optimization methods [41, 44, 11] to neural network-based methods [7, 29]. Despite good performance of convolutional neural networks (CNNs) over classical methods, a major drawback of CNNs is that they require tuning a large number of hyper-parameters and large training data sets. Blind deconvolution can be posed as an image-to-image translation problem where the blurred image is transformed into its matching unblurred image.

In this work, we use CGAN with a $l2$-contextual loss (squared difference between generated and target/sharp image) and an additional $l2$ high-frequency loss as regularization. This is motivated by the fact motion blur primarily affects image edges, a few discriminative image pixels compared to the entire image. The high-frequency images are first computed both for blurred and sharp images in the training data using iterative low pass-high pass filtering at 4 different scales [6]. These images are then used to provide additional information to the discriminator regarding the generator’s behavior (also see Fig. 5). Equation (3.4) becomes:

[TABLE]

where $\lambda=50$, $i=\left[0,1\right]$ refer to an original and high-frequency image pair and $l=2$. $x_{real}$ is the ground truth image for restoration (i.e. sharp images in our case). Minimization of Eq. (3.4) using Jensen-Shannon (JS) divergence as in [13] can lead to problems like mode collapse, vanishing gradients. Consequently, [3] proposed to use Wasserstein distance with gradient penalty (WGAN-GP). We propose thus to use CGAN with a critic network based on WGAN-GP [21]. The proposed model was trained for 300 epochs on a paired blur-sharp data set consisting of 10,710 (715 unique sharp images) multi-patient and multi-modal images with 15 different simulated motion trajectories for blur (see [21]).

3.4.2 Saturation or low contrast

The small or larger distances between the light source and the imaged tissue can lead to large illumination changes which can result in saturation or low contrast. This motivates the role of the variable $\gamma$ in Eq. (2) Saturated or low contrast image pixels often occur across large image areas compared to specularities and affect the entire image globally. In addition, these illumination changes are more prominently observed in normal brightfield (BF) modality compared to other modalities. Compensation of affected image pixels is a difficult problem depending on the size of the affected image area. We pose the saturation restoration task as an image-to-image translation problem and apply the same end-to-end CGAN approach used for motion deblur described above with $\mathit{l}_{2}$ contextual loss only to train a generator-discriminator network for saturation removal. Here, $\mathit{l}_{2}$ contextual loss is more suitable as we want to capture the deviation between normal illumination condition w.r.t saturation and low contrast conditions.

Due to lack of any ground truth data for two different illumination conditions, we created a fused data set that included: 200 natural scene images containing diffuse (scattered light) and ambient (additional illumination to natural light giving regions with pixel saturation) illuminations 111https://engineering.purdue.edu/RVL/Database/specularity_database/index.html; and 200 endoscopic image pairs simulated using cycleGAN-based style transfer [47] (separately trained on another 200 images with saturated and normal BF images with from 7 unique patients). To correct coloration shift due to the incorporation of natural images in our training set, color transfer (CRT) is applied to the generated frames. Given a source image, $I_{s}$ and a target image, $I_{t}$ to recolor, the mean $(\mathbf{\mu}_{s},\mathbf{\mu}_{t})$ and covariance matrix $(\mathbf{\Sigma}_{s},\mathbf{\Sigma}_{t})$ of the respective pixel values (in RGB channels) can be matched through a linear transformation [15]:

[TABLE]

where $I_{t}^{{}^{\prime}}$ is the recolored output. To avoid re-transfer of color from saturated pixel areas in the source, the mean and covariance matrix are computed from image intensities $<$90% of the maximum intensity value. Fig. 6 shows the generated results using our trained GAN-based network (on the right) and after color shift correction (bottom) showing very close to ground-truth results. To recover low contrast frames, the CGAN-saturation network was trained with a reverse image pair of the same training data set.

3.4.3 Specularity, and other misc. artifacts removal

Illumination inconsistencies and view point changes cause strong bright spots due to reflections from bubbles and shiny organ surfaces, and water-like substances can create multi-colored chromatic artifacts (referred to as ‘imaging or mixed artifact‘ in this paper). These inconsistencies appear as a combination of linear (e.g., additive noise $\eta$) and non-linear noise (function $F(.)$) in Eq. (2). A process referred to inpainting that uses the information of the surrounding pixels as prior information is used to replace the saturated pixels in affected regions. TV-inpainting methods are popular for restoring images with geometrical structures [37] and patch-based methods [10] for texture synthesis. However, these methods are computationally expensive. Recent advances in deep neural networks have proven to recover visually plausible image structures and textures [20] with almost real-time performance. However, they are limited to the size of the mask or the number of unknown pixels in an image. In this context, GANs [31, 16, 45] have been shown to be more successful in providing faster and more coherent reconstructions even with larger masks. Both contextual and generative losses have been used in these methods. Iizuka et al. [16] and Yu et al. [45] used local and global discriminators to improve the reconstruction quality. To enlarge the network receptive field [45] further used a coarse-to-fine network architecture using WGAN-GP instead of the DCGAN in [16]. Additionally, a $\mathit{l}_{1}$ discounted contextual (reconstruction) loss using a distance-based weight mask was used for added regularization [45]. Due to the reduced training time and better reconstruction quality compared to [16], we use the network proposed in [45] for inpainting.

We use a bottleneck approach to retrain the model initialised with the pretrained weights of the places2 data set [46]. To capture the large visual variations present in endoscopy images, 1000 images from 7 different patient endoscopy videos with a quality score $>$95% were used as the ‘clean’ images (see Section 3.3). We used 172 images as a validation set during the training. Both training and validation sets included multimodal endoscopic video frames. During training and validation masks of different patch sizes $\{(5\times 5),(7\times,7),(11\times 11),(13\times 13),...,(33\times 33)\}$ were randomly generated and were used for restoration. A single image can have one or multiple generated masks for restoration.

4 Experiments

4.1 Quality assessment metrics

To evaluate our artifact detection we use the standard mean average precision (mAP) and intersection-over-union (IoU) metrics. We quantitatively compare the detection results of all architectures using the mAP at IoU thresholds for a positive match of 5%, 25% and 50% denoted mAP5, mAP25 and mAP50 respectively, the mean IoU between positive matches, the number of predicted boxes relative to the number of annotated boxes and the average inference time for one image as quantitative measures. For the quality assessment of deblurring methods we use peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) measures. To overcome the limitations of PSNR for quantification of saturation and specularity restoration tasks we include more sophisticated visual information fidelity (VIF, [36]) and relative edge coherence (RECO, [5]) quality assessment metrics that are independent of the distortion type.

4.2 Artifact detection

Table 2 shows that YOLOv3 variants outperform both Faster R-CNN and Retinanet. YOLOv3-spp (proposed) yields the best mAP of 49.0 and 45.7 at IoU thresholds of 0.05 and 0.25 respectively at a detection speed $\approx 6\times$ faster than Faster R-CNN [34]. Even though Retinanet exhibits the best IoU of 38.9, it is to be noted that IoU is sensitive to annotator variances in bounding box annotation which might not resemble the performance of detectors. In terms of class-specific performance, from Fig. 7 and Table. 3 proposed YOLOv3-spp is the best across detecting misc. artifacts and bubbles (both are predominantly present in endoscopic videos) with average precision of 48.0 and 55.9, respectively. Faster R-CNN yielded the highest average precision for saturation (71.0) and blur (14.5) while RetinaNet and YOLOv3 outperformed respectively for contrast (73.6) and specularity detection (40.0). It is worth noting that proposed YOLOv3-spp yielded second best average precision scores for speculariy (34.7), saturation (55.7) and contrast (72.1).

4.3 Frame restoration

4.3.1 Blind deblurring

We compare our proposed conditional generative adversarial network with added contextual and high-frequency feature losses with deblurGAN [21], scale-recurrent network-based SRN-DeblurNet [39], and traditional TV-based method [11]. TV regularizion weight $\lambda$ and the blur kernel $r$ affects the quality of recovered deblurred images [11]. We chose $\lambda=10^{3}$ and $r=2.3$ after a few iterative parameter setting experiments for our data set. We performed retraining for SRN-DeblurNet [39] and deblurGAN [21] on the same data set used by our proposed deblurring model.

We quantitatively evaluated the frame deblurring methods using 5 images with visually large blur and our simulated test trajectories (see Table 4) and on 3 different test sequences (simulated motion blur trajectories, see Table 5) each with 30 images. Table 4 shows that CGAN with $l2$-contextual loss and added high-frequency (HF) feature loss score the highest PSNR and SSIM values for all blurred frames while TV-based deconvolution method [11] resulted in the least PSNR and SSIM values over all frames. Nearly, 1 dB increase can be seen against the deblurGAN method [21] for frames #80, #99, and #103 while $\approx$ 2 dB gain can be seen for #102, #116 against SRN-DeblurNet [39] using the proposed model. Overall the proposed model yields the best result compared to second best deblurGAN for the blurred image sequences in Table 5. This is also seen qualitatively in Fig. 8. It can be observed that SRN-DeblurNet deforms the image at upper right in both WL and NBI frames.

4.3.2 Saturation removal

We present results of treating for saturation removal as a global problem, correcting the entire frame for over exposure as discussed in Section 3.4.2. Quantitative results are provided in Table 6 for 19 randomly selected saturated frames from our simulated test data set derived from good quality frames (QS$>0.95$). Our restoration model demonstrates increased average values across all tested metrics, (PSNR, SSIM, VIF and RECO). Improvements after color transform for visual quality metrics like RECO (from 1.313 to 1.512), and VIF (from 0.810 to 0.818) illustrates boosted visual quality. This is also evident in qualitative result presented in Fig. 9. Largely saturated image patches in the left and central frames are clearly removed by the trained generator whilst preserving the underlying image details (see RGB histograms in Fig. 9, second row). The color transform successfully restores the original color consistency in CGAN restored images without introducing new saturation (see Fig. 9, last row). Note that simple contrast stretching as shown in Fig. 9 (third row) by rescaling the CGAN restored frames fails to recover the original color tones.

4.3.3 Specularity and other misc. artifacts removal

Specularity and other local artifacts are removed based on inpainting (Section 3.4.3 for details). To validate our inpainting methods, we used a set of 25 images (clean) with randomly selected patches covering 5% and 12% of the total pixels of $512\times 512$ image size. We compare our CGAN-based model with $l1$-contextual loss model with widely used traditional TV-based and patch-based inpainting methods. We observe in Table. 7 that $l1$-contextual CGAN method has the best quality assurance values for both VIF and RECO measures (VIF: 0.95, RECO:0.992 for 5% masked pixels and VIF: 0.883, RECO:0.983 for 12% masked pixels). Even though the TV-based inpainting method scored higher PSNR values in both cases, it scored the least RECO values (0.984 and 0.975 respectively for 5% and 12% cases) and has the highest computational cost (392 seconds). In contrast, $l1$-contexual CGAN has the least computational time (2s to load the trained model and apply on images on GeForce GTX 1080 Ti).

Qualitative results for our specularity and local artifact removal on real problematic gastro-oesophageal endoscopic frames are shown in Fig. 10. In Fig. 10 (a), both imaging artifacts (first and fourth rows) and specularities (second and third rows) introduce large deviations in pixel intensities both locally with respect to neighouring pixels and globally with respect to the uncorrupted image appearance. Using inpainting methods (see Fig. 10 (c) and (d)), the images have been restored based on the bounding box detections of our artifact detector. The second best TV-based method in Fig. 10(c) produces blurry and non-smooth patches during the reconstruction of unknown pixels (refer to Fig. 10 (b)) compared to CGAN generative model (see Fig. 10(d)). A closer look around the unknown regions indicated by blue rectangular boxes in Fig. 10(b), Fig. 10(e) shows that local image structures are well preserved and smoother transition from reconstructed pixels to the surrounding pixels is present. An immediate noticeable ghost effect can be observed in the second row, Fig. 10(e) top using the TV-based method.

4.4 Video recovery and quality assessment

We evaluated our artifact detection and recovery framework on 10 gastroesophageal videos comprising with nearly 10,000 frames each. For artifact detection, an objectness threshold of 0.25 was used to reduce duplication in detected boxes and QS value for restoring the frame was set to $\geq 0.5$. As a baseline, we also separately trained a sequential 6-layer convolution neural network (layer with 64 filters of sizes ${3,5\times 5}$, ReLU activation function and batch normalization) with a fully connected last layer for binary classification on a set of 6000 manually labeled positive and negative images to decide whether to discard or keep a given input video frame. A threshold of 0.75 was set for the binary classifier to keep only frames of sufficient quality. Our framework successfully retains the vast majority of frames compared to a binary decision, Fig.11. The quality enhanced video was again fed to our CNN-based binary classifier which resulted in lower number of frame rejection than on raw videos. Consequently, the resultant video is more continuous compared to the equivalent binary cleaned video utilizing raw videos. For example, in video 3, the video after frame removal based on the binary classifier directly lead to many distinct abrupt transitions that can be detrimental for post-processing algorithms as only 30% is kept. Comparatively, our proposed framework retains 70% of frames, i.e. a frame restoration of nearly 40%. Quantitatively across all 10 endoscopic videos tested, our framework restored 25% more video frames, retaining on an average of 68.7% of 10 videos considered.

4.5 Clinical relevance test

We corrupted 20 high-quality images selected from 10 test videos with blur, specularity, saturation and misc. artifacts (refer Sec. 3.4). Restoration methods were then applied to these images. Two expert endoscopists independently were asked to score these restoration results compared to the original high-quality images and corresponding videos. Scores (0-10) were based on: 1) Addition of unnatural distortions was assigned a negative score and 2) removal of distortions was assigned a positive score. The obtained mean score were blur: 7.87, specularity or misc. artifacts: 7.7, and saturation: 1.5. A remarkable restoration was obtained for blur and specularity or misc. artifacts. However, saturation correction was not pleasant to experts mostly due to loss of 3D information (according to feedback comments) even though visual coherence was improved.

5 Conclusion

We have presented a novel end-to-end framework for the detection and restoration of inevitable endoscopic video frame artifacts which are embodied in frames linearly, non-linearly or both. Our contribution includes artifact-specific quality assessment and sequential restoration. In particular, as each module in the proposed framework is formulated as a neural network, our framework can fully take advantage of the real-time processing capabilities of modern GPUs. We have proposed several novel techniques for frame restoration that includes an edge-based (high-frequency) loss for recovering blurred images and a color re-transfer scheme to deal with color shifts in generated frames due to model transfer. We have proposed novel regularization schemes based on each artifact class-type for frame restoration yielding high-quality image generation. Through extensive experiments we have validated each step of our framework, from detection to restoration methods. We achieved the highest $mAP_{5}$ and $mAP_{25}$ with our modulated YOLOv3-spp and the least inference time (88 ms) for real time frame quality scoring. We demonstrated quantitative and qualitative improvements for frame restoration tasks. Notably, we achieved improvements in both PSNR and SSIM metrics for blur and saturation using our proposed models. For specularity and other misc. artifacts removal, we achieved also significant improvements on visual similarity metrics. Finally, our sequential approach was able to restore an average of 25% of the video frames in 10 randomly selected videos from our database. It is worth noting that for 3 videos used for illustration of the importance of our proposed framework, 40% of frames which otherwise would be discarded for downstream analysis were rescued. We demonstrated high quality performance on real clinical endoscopy videos for both intra- and inter-patient variabilities and multimodality. Future work will focus on further improving the object detection network and implementing the entire framework as a single end-to-end trainable neural network.