Image Super Resolution via Bilinear Pooling: Application to Confocal Endomicroscopy

TL;DR

This paper introduces a novel attention mechanism combining first- and second-order statistics for super resolution of confocal endomicroscopy images, improving quality while maintaining efficiency for real-time clinical use.

Contribution

It proposes a new attention mechanism that integrates multiple statistical pooling methods, demonstrating competitive performance with fewer parameters.

Findings

Outperforms 11 existing super resolution methods on three datasets.

Achieves high PSNR, SSIM, and IFC metrics.

Is lightweight and suitable for real-time applications.

Abstract

Recent developments in image acquisition literature have miniaturized the confocal laser endomicroscopes to improve usability and flexibility of the apparatus in actual clinical settings. However, miniaturized devices collect less light and have fewer optical components, resulting in pixelation artifacts and low resolution images. Owing to the strength of deep networks, many supervised methods known as super resolution have achieved considerable success in restoring low resolution images by generating the missing high frequency details. In this work, we propose a novel attention mechanism that, for the first time, combines 1st- and 2nd-order statistics for pooling operation, in the spatial and channel-wise dimensions. We compare the efficacy of our method to 11 other existing single image super resolution techniques that compensate for the reduction in image quality caused by the necessity of endomicroscope miniaturization. All evaluations are carried out on three publicly available datasets. Experimental results show that our method can produce competitive results against state-of-the-art in terms of PSNR, SSIM, and IFC metrics. Additionally, our proposed method contains small number of parameters, which makes it lightweight and fast for real-time applications.

Tables

Table 1. Table 1: Details of the datasets used in our evaluation.

dataset	provided by	#patients	#images	anatomical site	image size
CLE100	Leong et al. [14]	30	181	small intestine	$1024\times 1024$
CLE200	Grisan et al. [6]	32	262	esophagus	$1024\times 1024$
CLE1000	Ştefănescu et al. [17]	11	1025	colorectal mucosa	$1024\times 1024$

Table 2. Table 2: Quantitative results of SR models at × 2 absent 2 \times 2 and × 4 absent 4 \times 4 factors. Bold indicates the best result. ✤ and ✝ denote traditional and DL-based methods, respectively. PSNR scores are reported with the standard error of the mean (SEM) for each method.

Methods		CLE100		CLE200		CLE1000		time
Scale $\times$2		PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic		33.69$\pm$0.06	0.8693	35.53$\pm$0.01	0.9029	34.45$\pm$0.01	0.8920	0.02
A+✤ [19]		34.22$\pm$0.07	0.8928	36.14$\pm$0.01	0.9218	35.04$\pm$0.01	0.9114	6.72
ANR✤ [18]		36.44$\pm$0.13	0.9226	39.10$\pm$0.01	0.9559	37.64$\pm$0.01	0.9559	6.07
GR✤ [18]		36.56$\pm$0.13	0.9243	39.26$\pm$0.01	0.9579	37.79$\pm$0.01	0.9448	4.47
SRCNN✝ [4]		35.75$\pm$0.11	0.9181	38.25$\pm$0.01	0.9494	36.87$\pm$0.01	0.9380	0.06
VDSR✝ [11]		36.72$\pm$0.13	0.9276	39.31$\pm$0.01	0.9578	37.89$\pm$0.01	0.9462	0.25
DRCN✝ [12]		36.65$\pm$0.13	0.9257	39.29$\pm$0.01	0.9575	37.83$\pm$0.01	0.9452	0.48
LapSRN✝ [13]		36.71$\pm$0.13	0.9264	39.25$\pm$0.01	0.9583	37.91$\pm$0.01	0.9462	0.07
SESR✝ [3]		36.76$\pm$0.13	0.9282	39.36$\pm$0.01	0.9583	37.91$\pm$0.01	0.9462	0.27
RBAM (Ours)✝		36.91$\pm$0.12	0.9321	39.45$\pm$0.01	0.9590	38.22$\pm$0.01	0.9501	0.18
Scale $\times$4
Bicubic		31.29$\pm$0.04	0.6673	32.45$\pm$0.01	0.7318	31.78$\pm$0.01	0.7278	0.02
A+ [19]		31.57$\pm$0.04	0.7042	32.76$\pm$0.01	0.7607	32.06$\pm$0.01	0.7517	3.03
ANR [18]		31.68$\pm$0.04	0.7160	32.93$\pm$0.01	0.7736	32.23$\pm$0.01	0.7671	2.88
GR [18]		31.70$\pm$0.04	0.7201	32.95$\pm$0.01	0.7736	32.25$\pm$0.01	0.7703	2.31
SRCNN [4]		31.59$\pm$0.04	0.7073	32.76$\pm$0.01	0.7617	32.07$\pm$0.01	0.7566	0.06
VDSR [11]		31.66$\pm$0.04	0.7144	32.86$\pm$0.01	0.7804	32.16$\pm$0.01	0.7635	0.25
DRCN [12]		31.70$\pm$0.04	0.7214	32.92$\pm$0.01	0.7750	32.21$\pm$0.01	0.7635	0.48
LapSRN [13]		31.68$\pm$0.04	0.7190	32.76$\pm$0.01	0.7617	32.29$\pm$0.01	0.7737	0.08
SESR [3]		31.76$\pm$0.04	0.7249	32.99$\pm$0.01	0.7804	32.29$\pm$0.01	0.7737	0.33
RBAM (Ours)		31.84$\pm$0.04	0.7315	33.11$\pm$0.01	0.7852	32.47$\pm$0.01	0.7874	0.07

Equations

$${\mathrm{H}}^0={\mathcal{F}}_{\mathrm{c}}({\mathrm{I}}_{\mathrm{LR}};{\theta }_0),$$

$${\mathrm{H}}^{\mathcal{B}}=\mathcal{B}({\mathrm{H}}^0)={\mathcal{G}}^{\mathrm{b}}({\mathcal{G}}^{\mathrm{b}-1}(...({\mathcal{G}}^1({\mathrm{H}}^0))...))+{\mathrm{H}}^0.$$

$${\mathrm{I}}_{\mathrm{SR}}={\mathcal{F}}_1(\mathcal{U}({\mathrm{H}}^{\mathcal{B}};{\theta }_{up});{\theta }_{rec}).\text{vspace}-3mm$$

$${\mathrm{H}}^{\mathrm{b}}={\mathcal{G}}^{\mathrm{b}}({\mathrm{H}}^{\mathrm{b}-1})={\mathcal{Q}}^{\mathrm{b}}({\mathrm{H}}^{\mathrm{b}-1})+{\mathrm{H}}^{\mathrm{b}-1},$$

$${\mathrm{H}}_{\mathrm{conv}}^{\mathrm{b}}={\mathcal{F}}_{\mathrm{c}}({\mathcal{F}}_{\mathrm{c}}({\mathrm{H}}^{\mathrm{b}-1};{\theta }_1^b);{\theta }_2^b).$$

$${\mathrm{H}}_{\mathrm{ca}}^{\mathrm{b}}={\mathrm{H}}_{\mathrm{conv}}^{\mathrm{b}}\otimes \sigma ({\mathcal{F}}_{\mathrm{c}}({\mathcal{F}}_{\frac{\mathrm{c}}{4}}({\mathrm{F}}_{\mathrm{ca}}^{1st}+{\mathrm{F}}_{\mathrm{ca}}^{2nd};{\theta }_3^b);{\theta }_4^b)).$$

$${\mathrm{H}}_{\mathrm{sa}}^{\mathrm{b}}={\mathrm{H}}_{\mathrm{conv}}^{\mathrm{b}}\otimes \sigma ({\mathcal{F}}_{\mathrm{c}}({\mathrm{F}}_{\mathrm{sa}}^{1st}+{\mathrm{F}}_{\mathrm{sa}}^{2nd};{\theta }_5^b))$$

$${\mathrm{H}}^{\mathrm{b}}={\mathcal{F}}_{\mathrm{c}}({\mathrm{H}}_{\mathrm{ca}}^{\mathrm{b}}++{\mathrm{H}}_{\mathrm{sa}}^{\mathrm{b}};{\theta }_6^b)+{\mathrm{H}}^{\mathrm{b}-1}.$$

Full text

1 Introduction

Colorectal cancer is known as the fourth most-common cancer and remains one of the leading causes of cancer related mortality in the world. In 2018, more than 1 million people were affected by colorectal cancer worldwide, resulting in an estimated 550,000 deaths [2]. Rapid histopathologic assessment is an important tool that may improve disease prognosis by detecting early-stage cancer and pre-cancerous conditions. Although biopsy and ex-vivo tissue examination are widely accepted as the diagnostic gold standard, such procedures take time and may limit the ability of the endoscopist to rapidly gauge disease severity. Confocal laser endomicroscopy (CLE), on the other hand, has substantially improved real-time in-vivo visualization of the subsurface of living cells, vascular structures, and tissue patterns during endoscopic examination [10].

For in-vivo histological examination, the large size of the microscope complicates navigation of the interior of the body in a clinical setting. Therefore, it is necessary to reduce the size of the microscope to completely and safely access the organ(s) of interest. However, miniaturization reduces the number of optical elements in the microscope probe, introducing pixelation artifacts in the acquired images. One strategy to remove image artifacts and enhance image quality is to directly post-process degraded images. An emerging process in the field of image processing, referred to as single image super-resolution (SR), aims to reconstruct an accurate high-resolution (HR) image given its low-resolution (LR) counterpart. Thus, SR is a promising software method to mitigate image degradation due to hardware miniaturization.

Among traditional SR algorithms, Huang et al. [8] proposed leveraging self-similarity modulo affine transformations to accommodate natural deformation of recurring statistical priors within and across scales of an image. Timofte et al. [18, 19] used a combination of neighbour embedding and sparse dictionary learning over an external database and proposed anchored neighborhood regression in the dictionary atom space. Recently, CNNs have advanced the SR research field by directly learning the mapping between LR and HR images [4, 11, 12, 13, 1]. Dong et al. [4] demonstrated that a fully convolutional network trained end-to-end can perform LR-to-HR nonlinear mapping. Kim et al. [11] suggested a trained network to predict additive details in the form of a residual image, which is summed with the interpolated image. Kim et al. [12] addressed model overfitting by reducing the number of parameters via recursive convolutional layers. Lai et al. [13] designed a network which progressively reconstructs the sub-band residuals of high-resolution images at multiple pyramid levels. Ahn et al. [1] improved speed and efficiency of SR models by designing a cascade mechanism over residual networks. Lastly, Cheng et al. [3] exploited recursive squeeze and excitation modules in a network to exploit relationships between channels. Izadi et al. [9] reported the first attempt to deploy CNNs on CLE images. They used a densely connected CNN to transform synthetic LR images into HR ones. Ravi et al. [15] employed a CNN to restore missing details into LR images. They collected a set of consecutive LR frames and generated synthetic HR images using a video registration technique. In a more recent study [16], Ravi et al. trained a CNN for unsupervised SR on CLE images using a cycle consistency regularization, designed to impose acquisition properties on the super-resolved images.

In this paper, we present a lightweight convolutional neural network (CNN) that is appropriate for frame-wise SR by incorporating a novel attention mechanism. In contrast to SESR [3], which leverages attention modules from the Squeeze-and-Excitation network (SENet) [7] to re-weight channels, we introduce a novel weighting scheme to recalibrate learned features based on pairwise relationships. Our attention modules compromise both 1${}^{\text{st}}$-order pooling and 2${}^{\text{nd}}$-order pooling (a.k.a. bilinear pooling), improving the quality of learned features in the network by considering pairwise correlations along feature channels and spatial regions [5]. The compactness and computational speed of our network lends well to real-time implementation during in-vivo examination. We demonstrate that stacking attention modules in the middle of a low-level feature extraction head and a feature integration tail quantitatively and qualitatively produces superior results against existing SR methods and generalizes well over unseen microscopic datasets.

2 Method

Network Overview. Fig. 1-a depicts the overall architecture of our proposed LR-to-SR network. Let $\mathrm{L}_{\mathrm{LR}}\in\mathbb{R}^{1\times W\times H}$, $\mathrm{I}_{\mathrm{SR}}\in\mathbb{R}^{1\times rH\times rW}$, and $r$ denote the low resolution input and super-resolved output, and the downsample factor, respectively. We use a convolution layer, denoted by $\mathcal{F}(\cdot)$, with a $3\times 3$ kernel and $C$ output channels to extract initial features $\mathrm{H}^{\mathrm{0}}\in\mathbb{R}^{C\times H\times W}$ , i.e.

[TABLE]

where $\theta$ refers to the learnable parameters. In our proposed network, the initial features $\mathrm{H}^{\mathrm{0}}$ are updated by sequential residual attention modules, denoted as $\mathcal{G(\cdot)}$ and a skip connection. The entire high-level feature extraction stage is denoted as $\mathcal{B}(\cdot)$:

[TABLE]

To upsample the feature maps, we use sub-pixel convolutions, denoted as $\mathcal{U}(\cdot)$, followed by a single channel $1\times 1$ convolution for SR reconstruction:

[TABLE]

Residual Bilinear Attention Module. In our proposed RBAM, we combine 1${}^{\text{st}}$- and 2${}^{\text{nd}}$-order pooling operations spatially and channel-wise to recalibrate learned features for efficient network training. Fig. 1-b illustrates the structure of our proposed RBAM. Mathematically, we formulate RBAM as:

[TABLE]

where $\mathcal{Q}(\cdot)$ denotes the attention modules before the skip connection. Given the input feature maps $\mathrm{H^{\mathrm{b}}}\in\mathbb{R}^{C\times H\times W}$, two convolutions with $3\times 3$ kernel size interleaved with a ReLU activation function are performed to produce high-level feature maps $\mathrm{H^{\mathrm{b}}_{\mathrm{conv}}}\in\mathbb{R}^{C\times H\times W}$ as input to the attention branches:

[TABLE]

Channel-wise Attention (CA) Branch. CA leverages the inter-channel correspondence between feature responses (Fig. 1-c). 1${}^{\text{st}}$- and 2${}^{\text{nd}}$-order pooling mechanisms operate on $\mathrm{H}^{\mathrm{b}}_{\mathrm{conv}}$, producing two vectors $\mathrm{F_{\mathrm{ca}}^{\mathrm{1st}}}$, $\mathrm{F_{\mathrm{ca}}^{\mathrm{2nd}}}\in\mathbb{R}^{C\times 1\times 1}$. $\mathrm{F_{\mathrm{ca}}^{\mathrm{1st}}}$ is the 1${}^{\text{st}}$-order CA obtained by spatial average pooling to squeeze the feature map of each channel [7]. To obtain 2${}^{\text{nd}}$-order CA, pairwise channel correlations are computed in the form of a covariance matrix $\mathrm{\mathrm{\Sigma}}\in\mathbb{R}^{C\times C}$ by spatial flattening, dimension permutation, and matrix multiplication. Each row in $\mathrm{\Sigma}$ encodes the statistical dependency of a channel with respect to every other channel [5]. Given the covariance matrix $\mathrm{\Sigma}$, we adopt a row-wise convolution with $1\times C$ kernel size to produce the 2${}^{\text{nd}}$-order CA vector $\mathrm{F_{\mathrm{ca}}^{\mathrm{2nd}}}$. Finally, two successive 1-D convolutions interleaved with a ReLU activation function operate on a vector formed by the sum of $\mathrm{F_{\mathrm{ca}}^{\mathrm{1st}}}+\mathrm{F_{\mathrm{ca}}^{\mathrm{2nd}}}$. The output of the convolution operation is fed into a sigmoid function $\sigma$, followed by element-wise multiplication $\otimes$ to produce the bth updated features maps $\mathrm{H}_{\mathrm{ca}}^{\mathrm{b}}$:

[TABLE]

Spatial Attention (SA) Branch. SA indicates shared correspondence between spatial regions across all feature maps (Fig. 1-d). Given $\mathrm{H}^{\mathrm{b}}_{\mathrm{conv}}$ as the input, the 1${}^{\text{st}}$-order spatial attention matrix, $\mathrm{F}_{\mathrm{sa}}^{\mathrm{1st}}\in\mathbb{R}^{1\times H\times W}$, is computed by the average pooling operation along channel dimension to aggregate information for each spatial location across all features. To compute 2${}^{\text{nd}}$-order spatial attention matrix, $\mathrm{F}_{\mathrm{sa}}^{\mathrm{2nd}}\in\mathbb{R}^{1\times H\times W}$, we first reduce the spatial size of feature maps to $H^{\prime}\times W^{\prime}$ ($8\times 8$ in our implementation) by applying average pooling. Then, appropriate reshaping, dimension permutation and matrix multiplication is adopted to obtain the covariance matrix $\mathrm{\Sigma}\in\mathbb{R}^{H^{\prime}W^{\prime}\times H^{\prime}W^{\prime}}$. Similar to channel-wise attention, a row-wise convolution with $1\times H^{\prime}W^{\prime}$ kernel size is applied on $\mathrm{\Sigma}$. Eventually, dimension permutation and nearest neighbor interpolation produce $\mathrm{F}_{\mathrm{sa}}^{\mathrm{2nd}}$. We add these two matrices together element-wise and apply a convolution with $1\times 1$ kernel size that feeds a sigmoid function. Spatial attention is realised by element-wise multiplication over all feature maps, formulated as:

[TABLE]

Attention Fusion. The updated features are concatenated ($\mathbin{+\mkern-10.0mu+}$) and aggregated via a convolution with kernel $1\times 1$ kernel. Lastly, $\mathrm{H}^{\mathrm{b}}$ is added via skip connection:

[TABLE]

3 Results and Discussion

Data. We evaluate existing state-of-the-art SR methods, as well as our proposed RBAM, on three publicly available CLE datasets (Table 1). We select images rich in texture by assessing the SR performance of bicubic interpolation on the unseen test set. As depicted in Fig. 3, images with PSNR scores below the mean PSNR score of the bicubic method evalulated on the test set are deemed ’texture rich’, and are used for evaluation, whereas images associated with scores above the mean are deemed ’texture poor’. In other words, images which can be effectively restored using bicubic interpolation are rejected for evaluation, as they contain little information on which to assess the performance of state-of-the-art methods. Evaluation assesses the methods’ ability to reconstruct $1024\times 1024$ HR image from a synthesized LR counterpart obtained via bicubic downsampling with the appropriate factor ($\times$2 or $\times$4).

Training Settings. We train all methods on a random partition (80%) of CLE100, and evaluate them on the remaining 20% as well as CLE200, and CLE1000. For DL-based methods, we replicated the reported training settings, and used public code for traditional algorithms. For our model, we use $B=5$ RBAMs and set the number of features to $C=64$ to create a lightweight network. In each training batch, 16 LR patches of size $48\times 48$ are randomly extracted as inputs, and augmented by random 90*∘* rotations and horizontal/vertical flip. We use Adam optimizer and L1 loss to train our network for 300 epochs. Initial learning rate is set to $10^{-4}$ and is halved every 50 epochs.

Ablation Investigation. We discern the effectiveness of the individual components in our network modules by ablating attention blocks and evaluating performance after 50 epochs. Our investigation shows that, for CLE100 at $\times 2$ SR, attention-based variants outperform the baseline, demonstrating the merits of incorporating spatial and channel-wise contextual information. We also observed that using both 1${}^{\text{st}}$ and 2${}^{\text{nd}}$-order pooling operations simultaneously outperform using either 1${}^{\text{st}}$ or 2${}^{\text{nd}}$-order channel-wise pooling individually. We similarly note that using both spatial and channel-wise attention outperforms either one alone.

Comparison to State-of-the-art. We compare the performance of traditional algorithms including ANR [18], GR [18] and A+ [19], as well as DL-based techniques including SRCNN [4], VDSR [11], DRCN [12], LapSRN [13], SESR [3] and our proposed RBAM. Table 2 summarizes the quantitative comparisons in terms of peak signal-to-noise-ratio (PSNR-SEM), structural similarity (SSIM), and inference time at $\times 2$, and $\times 4$ SR. From the table, one can see that most DL-based methods consistently outperform traditional SR algorithms in PSNR and SSIM metrics. Particularly, RBAM significantly outperforms the mean PSNR over all datasets by 0.18dB and 0.13dB for $\times 2$ and $\times 4$ SR, respectively. Furthermore, RBAM is a practical compromise between inference time, and generalization. Our results show a moderate quantitative increase in PSNR score and a considerable increase in qualitative performance - this is similar to previous works in single image super resolution [20]. Fig. 2 shows selected image patches from each dataset for qualitative assessment. RBAM can delicately restore high-frequency cues, such as granular textures and sudden changes in grayscale pixel intensity. This manifests qualitatively in the form of improved restoration of high frequency details such as cell membranes (CLE200, CLE1000 examples) and intracellular spaces (CLE100 example).

Motivation for Bilinear Pooling. We combine 1${}^{\text{st}}$-order and 2${}^{\text{nd}}$-order pooling to recalibrate learned features based on channels that activate often or correspond to feature rich inputs, respectively. Channels that activate often are likely responding to common, low frequency image features. Conversely, channels that are highly correlated may be responding to feature rich instances in the image space that activate multiple filters simultaneously. High frequency features tend to be complex, and not as common semantically compared to low frequency image details. Therefore, channels that learn complex image features may not be emphasized by first order pooling operations alone. Combining first and second order pooling in an attention module assures that hard working channels are rewarded without diminishing the optimization of channels that learn complex features in the low to high resolution image mapping space.

4 Conclusion

We proposed the first network that simultaneously leverages both first and second order statistics for pooling in spatial and channel-wise attention mechanisms, resulting in a lightweight and fast model that restores high frequency image details. We compared our proposed model with various traditional and DL-based SR techniques on three CLE datasets in terms of image quality assessment metrics and inference time. Our RBAM network outperforms existing lightweight methods across different datasets, downsampling factors, and SR performance evaluation criteria. Experimental results also highlight the potential applicability of inexpensive software-based post-processing SR modules that improve degraded images in miniaturized CLE devices in real-time.

Acknowledgments. Thanks to the NVIDIA Corporation for the donation of Titan X GPUs used in this research and to the Collaborative Health Research Projects (CHRP) for funding.