WhiteNNer-Blind Image Denoising via Noise Whiteness Priors

TL;DR

This paper introduces a neural network model that leverages noise whiteness priors and image regularization techniques to effectively denoise medical images without requiring ground truth data, outperforming existing methods.

Contribution

The novel approach combines noise whiteness priors with traditional image priors in a neural network framework for blind image denoising without ground truth.

Findings

Outperforms Noise2Noise and Noise2Self in PSNR and SSIM

Effective on confocal laser endomicroscopy datasets

No ground truth data needed for training

Abstract

The accuracy of medical imaging-based diagnostics is directly impacted by the quality of the collected images. A passive approach to improve image quality is one that lags behind improvements in imaging hardware, awaiting better sensor technology of acquisition devices. An alternative, active strategy is to utilize prior knowledge of the imaging system to directly post-process and improve the acquired images. Traditionally, priors about the image properties are taken into account to restrict the solution space. However, few techniques exploit the prior about the noise properties. In this paper, we propose a neural network-based model for disentangling the signal and noise components of an input noisy image, without the need for any ground truth training data. We design a unified loss function that encodes priors about signal as well as noise estimate in the form of regularization terms. Specifically, by using total variation and piecewise constancy priors along with noise whiteness priors such as auto-correlation and stationary losses, our network learns to decouple an input noisy image into the underlying signal and noise components. We compare our proposed method to Noise2Noise and Noise2Self, as well as non-local mean and BM3D, on three public confocal laser endomicroscopy datasets. Experimental results demonstrate the superiority of our network compared to state-of-the-art in terms of PSNR and SSIM.

Tables

Table 1. Table 1 : Summary of datasets used in our experiments.

\rowcolor[rgb] .851, .851, .851 Dataset	provided by	#patients	#images	anatomical site	image size
CLE100	Leong et al. [19]	30	181	small intestine	$1024\times 1024$
CLE200	Grisan et al. [11]	32	262	esophagus	$1024\times 1024$
CLE1000	Ştefănescu et al. [26]	11	1025	colorectal mucosa	$1024\times 1024$

Table 2. Table 2 : Quantitative results of WhiteNNer against baseline methods in term of PSNR and SSIM on held-out test data from three CLE datasets. Columns denoted by σ 𝜎 \sigma , 𝒮 𝒮 \mathcal{S} and 𝒩 ′ superscript 𝒩 ′ \mathcal{N}^{\prime} indicate whether methods leverage noise variance, ground truth, or different realization of the noisy image. Error bars for CNNs is calculated from training five models.

\rowcolor[rgb] .851, .851, .851                                                                                                           Dataset

CLE100

CLE200

CLE1000

\rowcolor[rgb] .851, .851, .851                                                                                                     Metric

$\sigma$

$𝒮$

${𝒩}^{\prime }$

PSNR$\uparrow$

SSIM$\uparrow$

PSNR$\uparrow$

SSIM$\uparrow$

PSNR$\uparrow$

SSIM$\uparrow$

NLM   [5]

✓

✕

✕

23.49

0.3236

22.87

0.3318

23.40

0.3383

BM3D   [7]

✓

✕

✕

25.91

0.4152

24.74

0.4297

25.58

0.3877

N2S   [2]

✕

✕

✕

26.75$\pm$0.533

0.4467

25.12$\pm$0.521

0.4817

25.99$\pm$0.450

0.3989

N2N   [17]

✕

✕

✓

28.04$\pm$0.396

0.5070

26.00$\pm$0.297

0.5699

27.97$\pm$0.149

0.5139

N2T

✕

✓

✕

28.31$\pm$0.268

0.5128

26.79$\pm$0.495

0.5790

28.29$\pm$0.362

0.5315

WhiteNNer-1

✕

✕

✕

25.75$\pm$0.275

0.4171

24.62$\pm$0.127

0.4627

25.63$\pm$0.401

0.3934

WhiteNNer-2

✕

✕

✕

27.05$\pm$0.184

0.4895

25.79$\pm$0.275

0.5301

26.58$\pm$0.245

0.4694

Table 3. Table 3 : Paired t-test on the CLE100 dataset at σ = 50 𝜎 50 \sigma=50 .

\rowcolor[rgb] .851, .851, .851                               Method			$p$-value	$t$-value
NLM   [5]	1.35e-16	12.83
BM3D   [7]	1.25e-12	9.44
Noise2Self   [2]	2.69e-3	3.11

Table 4. Table 4 : Performance of different loss function in terms of average PSNR. The results are reported on the CLE100 dataset at σ = 50 𝜎 50 \sigma=50

\rowcolor[rgb] .851, .851, .851                        Loss		PSNR$\uparrow$
${ℒ}_{rec}+{ℒ}_{pc}$	22.41
${ℒ}_{rec}+{ℒ}_{pc}+{ℒ}_{tv}$	23.14
${ℒ}_{rec}+{ℒ}_{pc}+{ℒ}_{tv}+{ℒ}_{ar}$	25.88
${ℒ}_{rec}+{ℒ}_{pc}+{ℒ}_{tv}+{ℒ}_{st}$	26.02
${ℒ}_{rec}+{ℒ}_{pc}+{ℒ}_{tv}+{ℒ}_{ar}+{ℒ}_{st}$	26.12

Equations

$$x_i=s_i+n_i$$

$$n_i\backsim \mathrm{N}(0,\sigma )$$

$${\mathcal{L}}_{total}(\mathcal{X},\mathcal{S},\mathcal{N})={\mathcal{L}}_{rec}(\mathcal{X},\mathcal{S},\mathcal{N})+{\mathcal{L}}_{noise}(\mathcal{N})+{\mathcal{L}}_{signal}(\mathcal{S})$$

$${\mathcal{L}}_{noise}(\mathcal{N})={\mathcal{L}}_{ac}(\mathcal{N})+{\mathcal{L}}_{st}(\mathcal{N})$$

$${\mathcal{L}}_{signal}(\mathcal{S})={\mathcal{L}}_{pc}(\mathcal{S})+{\mathcal{L}}_{tv}(\mathcal{S})$$

$${\mathcal{L}}_{rec}(\mathcal{X},\mathcal{S},\mathcal{N})=\frac{1}{M}\sum _{i=1}^M[x_i-(s_i+n_i){]}^2$$

$$\mathcal{R}(\tau )=E[y(t)\cdot y(t-\tau )]$$

$$\mathcal{R}(\ell ,m)=E[I(i,j)I(i-\ell ,j-m)]$$

$$\mathcal{R}(\ell,m)=\left\{\begin{array}[]{ll}{\sigma^{2}}&{\text{ if }(l,m)=(0,0)}\\ {0}&{\text{ otherwise }}\end{array}\right.$$

$$\bar{R}(\ell )=\frac{1}{M}\sum _{i=1}^M{n}_i\cdot n_{i+\ell }$$

$$Var[n_i]=Var[n_{i+\ell }]$$

$${\hat{\sigma }}_i={\hat{\sigma }}_j;\forall i,j\in \{1,2,...,B\}$$

$${\mathcal{L}}_{pc}=\frac{1}{M}\sum _{i=1}^M[{\mathcal{G}}_x(s_i)-{\mathcal{G}}_x(m_i){]}^2+[{\mathcal{G}}_y(s_i)-{\mathcal{G}}_y(m_i){]}^2$$

$${\mathcal{L}}_{tv}=\frac{1}{M}\sum _{i=1}^M|{\mathcal{G}}_x(s_i)|+|{\mathcal{G}}_y(s_i)|$$

Full text

1 Introduction

Globally, colorectal cancer is the third and the second most commonly diagnosed cancer in men and women, respectively. In 2018, 575,789 men and 520,812 women had a history of colorectal cancer, resulting in an estimated 551,269 deaths [3]. Timely inspection of suspicious areas within the gut, followed by a precise diagnosis, is critical for improved disease prognosis and reduced mortality. However, conventional tissue sectioning and ex-vivo histological examination are associated with invasive biopsy collection and preparation that significantly delay the screening procedure.

Handheld, portable confocal laser endomicroscopy (CLE) is a well-established imaging technique that provides a real-time, in-vivo, and biopsy-free histological assessment (so-called optical biopsy) of the mucosal layer of the gastrointestinal (GI) tract based on both endoscopic and endomicroscopic images acquired during an ongoing endoscopy. CLE provides magnified visualization of tissues in cellular and subcellular resolution and enables the endoscopists to assess the pathology in gastrointestinal tissue sites, e.g., Barrett’s esophagus [15] or colonic mucosa [14].

As the accuracy of medical imaging-based diagnostics depends on the quality of the images, it is critical to curtail imaging noise. One approach for noise reduction is to rely on higher quality image acquisition, be that through longer acquisition times, more elaborate optics, high fidelity electronics, or more complex image reconstruction algorithms. However, all these strategies are not without their drawbacks, i.e., subjecting the patient to lengthier procedures, less portable and more invasive imaging devices, increased cost, or lower frame rates, respectively. An alternative strategy to enhance the image quality is to directly post-process the corrupted images.

Due to the ill-posed nature of the image denoising problem, a large body of classic methods utilizes different regularization techniques, like incorporating additional domain-specific prior knowledge in the model to restrict the possible candidates in the optimization search space by penalizing solutions deemed undesirable [23, 7, 8, 5]. Despite the advantage of leveraging priors about the properties of the true signal, exploiting available information about the noise is often completely ignored. Conversely, the estimate of the clean image can be further improved once we also ensure that the noise estimate conforms to the known and expected properties of the true imaging noise as well.

Recently, convolutional neural networks (CNNs) have shown remarkable performance for image restoration, particularly image denoising [29, 30, 27]. These CNN-based approaches are typically trained in a supervised manner, requiring pairs of noisy and clean images (ground truth). However, in many applications, it is practically impossible to acquire the ground truth images, e.g., medical imaging of real patients. Lehtinen et al. [17] proposed the Noise2Noise training scheme to relax the requirement of providing ground truth by allowing the network to learn the mapping between two instances of the noisy image containing the same signal. Despite the efforts to remove the need for clean images in the training procedure, it is unfortunately still impractical to acquire several instances of the noisy image in some contexts, particularly for endomicroscopy. Alternatively, Batson et al. [2] proposed Noise2Self which is a self-supervised approach to denoise a corrupted image from only a single noisy instance.

In this paper, in addition to relying on explicit signal priors, we propose incorporating novel regularization terms into the CNN loss function thus encouraging the noise to respect the whiteness priors. Using these priors, we train our network to map the noisy images to the clean and noise components, without any ground truth. Moreover, instead of computing the noise by subtracting the predicted signal from the noisy input, we employ a neural architecture with two tails – one for inferring the clean image and the other for the noise. This design can serve as a new architectural regularization to facilitate the separability of signal and noise in a latent space, leading to the better reconstruction of both.

In a nutshell, the main contributions of the paper are summarized as follows:

$\bullet$

We propose a novel blind denoising model that leverages both signal and noise priors in the form of regularization terms in a CNN loss function.

$\bullet$

We present the first network that is capable of decoupling signal and noise without any ground truth.

$\bullet$

We demonstrate that our proposed method outperforms state-of-the-art blind denoising techniques on three confocal endomicroscopy datasets.

2 Related Works

Most existing image denoising techniques in the literature leverage prior knowledge and we categorize them into the following three general groups:

Data-driven priors. Instead of explicit consideration, the approaches in this group encode the priors implicitly through learning the direct mapping between pairs of corrupted and clean images. Thanks to the strength of deep networks, recent denoising methods have achieved impressive results. The first attempt to leverage neural networks for image denoising was conducted by Burger et al. [6] who used a simple multi-layer perceptron architecture for the learning. Afterward, Zhang et al. [29] demonstrated that residual learning and batch normalization not only improve the denoising performance but also expedite the training procedure. Thai et al. [27] proposed a very deep memory persistent network by stacking dozens of memory units that are densely connected. Zhang et al. [30] suggested feeding a noise level map as well as the noisy image to the network. Doing so, the network will be able to handle different levels of corruption and spatially-variant noise. Most recently, the network proposed by Guo et al. [12] consists of one noise level estimator and one non-blind denoiser sub-network to robustly generalize the model performance on real-world noise.

Signal Priors. Approaches in this group, which encompasses the majority of the traditional image denoising techniques, exploit the hand-crafted priors about the underlying signal, such as smoothness [4], gradient [23], sparsity [9] and non-local self-similarity [7]. Classic approaches for image denoising can be traced back to the piecewise constant prior in the Mumford-Shah model, which was originally proposed for image segmentation [21], and then extended by Tsai tet al. [28] for image denoising. Later, Rudin et al. [23] exploited total variation (TV) prior in a variational formulation to preserve sharp edges of the underlying signal while suppressing the noise. Non-local self-similarity (NSS) prior relates to the fact that images often contain many similar yet non-local (i.e. spatially distant) patterns within the image. BM3D [7] is an NNS-based denoising method which groups similar patches, transforms the groups into the frequency domain and attenuates the noise by hard-thresholding of the transform coefficients. More recently, Ulyanov et al. [18] proposed to leverage the deep networks with randomly-initialized weights as a hand-crafted prior for image denoising. In their method, the network is trained to reconstruct the corrupted image from random noise input with an early stop constraint to prevent fitting to the noise. One can interpret their work as an NNS-based algorithm since a linear combination of spatially shared kernels is used to generate a clean image.

Noise Priors. There have been several attempts to exploit prior knowledge about spectral whiteness of the residual images in deblurring problem [1, 13, 24]. However, to the best of our knowledge, only a few have focused on image denoising. Lanza et al. [16] proposed to explicitly enforce whiteness property by imposing soft constraints on the auto-correlation of the residual image in the frequency domain. Recently, Soltanayev  [25] leveraged the Gaussianity property of the AWGN to adopt Stein’s Unbiased Risk Estimator (SURE) as a loss function for training the deep networks without clean ground truth images.

3 Method

Image denoising refers to the process of inspecting the noisy image to decouple the underlying signal from the noise component. One common assumption is that the noise is additive white Gaussian (AWGN) with a standard deviation of $\sigma$. Let us consider $\mathcal{X}=\{x_{i}\in\mathbb{R}\}_{i=1}^{M}$ to represent the noisy image formed by adding random white Gaussian noise $\mathcal{N}=\{n_{i}\in\mathbb{R}\}_{i=1}^{M}$ to a clean signal $\mathcal{S}=\{s_{i}\in\mathbb{R}\}_{i=1}^{M}$ where $M$ denotes total number of pixels. Letting $i$ to denote the index for a single pixel, the degradation model can be written as follows:

[TABLE]

[TABLE]

The goal of our model is to take a single noisy image and decouple it back into the signal $\mathcal{S}$ and noise $\mathcal{N}$, without any ground truth. To do so, we need to design accurate and discriminative priors for either component and use them as the supervisory signal during the training.

3.1 Architecture and Inference

As depicted in Fig 2, our architecture embodies the encoder-decoder paradigm with skip connections [22] (removed in Fig 2 for simplicity). The encoder $\phi_{enc}$ takes the noisy image $\mathcal{X}$ where each pixel is normalized to $[0,1]$ and outputs two $p$-dimensional latent features $\mathcal{F}^{\mathcal{N}}=\{f_{i}^{n}\in\mathbb{R}\}_{i=1}^{p}$, $\mathcal{F}^{\mathcal{S}}=\{f^{s}_{i}\in\mathbb{R}\}_{i=1}^{p}$, The features $\mathcal{F}^{\mathcal{S}}$ and $\mathcal{F}^{\mathcal{N}}$ are then decoded to the spatial domain by a shared decoder $\phi_{dec}$, resulting in signal $\mathcal{S}$ and noise $\mathcal{N}$ estimates. Accordingly, our network has a single input and two outputs. Once the model is trained, the latent representation for the noise $\mathcal{F}^{\mathcal{N}}$ can be simply discarded as it is only provided to serve as an architectural regularizer. In other words, the models maps the $\mathcal{X}$ to both $\mathcal{F}^{\mathcal{S}}$ and $\mathcal{F}^{\mathcal{N}}$ while only $\mathcal{F}^{\mathcal{S}}$ is fed into the decoder to reconstruct signal $\mathcal{S}$.

We use the original U-Net [22] architecture for all our experiments with depth 5, kernel size 3, bilinear upsampling and linear activation in the last layer.

3.2 Proposed Loss Functions

Our model adopts a multi-loss objective function which encodes reconstruction, a.k.a data fidelity, and priors about the properties of the signal and noise. The proposed loss function can be formulated as follows:

[TABLE]

The noise prior $\mathcal{L}_{noise}$ consists of two distinct terms. The first term, auto-correlation loss $\mathcal{L}_{ac}$, leverages the statistical fact that a white noise image contains pixels intensities which are spatially uncorrelated, given the signal. The second term, stationary loss $\mathcal{L}_{st}$, penalizes the network if the variance of the noise estimate is not consistent spatially. Hence, the loss term for noise priors can be written as:

[TABLE]

Turning to signal properties, we take two well-established priors into consideration: the piecewise constancy [21] and minimal total variation [23] denoted by $\mathcal{L}_{pc}$ and $\mathcal{L}_{tv}$, respectively. Therefore, signal prior can be written as follows:

[TABLE]

Reconstruction Loss. Given the signal and noise estimates $\mathcal{S}$ and $\mathcal{N}$ in the output, $\mathcal{L}_{rec}$ guarantees that the addition of these two components perfectly reconstructs the original input. We use $L_{2}$ distance to measure the faithfulness over all pixels. Mathematically,

[TABLE]

Auto-correlation Loss. In statistics, auto-correlation (AC) is the metric to measure the correlation or similarity between a random process and a time-lagged version of itself. For a discrete random process $y$, the AC at lag $\tau$ is expressed as:

[TABLE]

In context of 2D images, the lag can be considered in spatial domain across horizontal and vertical directions. Let $I(i,j)$ denote an image in 2D space where $i$ and $j$ are the pixel coordinates. We can define the AC function as a mapping from pixel coordinates to a scalar value, formulated as follows:

[TABLE]

where $\ell$ and $m$ are the spatial lag between two distinct pixel coordinates across the horizontal and vertical axis, respectively. On the other hand, the whiteness property of the noise implies every pixel to be uncorrelated to any other pixel. Alternatively stated, given a white noise image, the AC at lag $\ell=0$ and $m=0$ equals to the noise variance $\sigma^{2}$ while being zero elsewhere, i.e.:

[TABLE]

Moreover, we assume that the noise is ergodic and the image to be denoised is sufficiently large. Following our notation, sample auto-correlation can be used to estimate the AC for a noise image $\mathcal{N}$, defined as:

[TABLE]

To implement $\mathcal{L}_{ac}$, we use sample AC and minimize it for any lag value greater than zero to penalize noise estimates that are spatially correlated. Precisely, we first pad each side of the noise prediction with a reflected copy of itself. Then, the sample AC is calculated by selecting a random lag in the range of $\ell\in[1,M]$ for each update.

Stationary Loss. In general, a random process is said to be stationary if its statistics are not changed over time. White noise image is a simple example of a stationary process where the variance and mean are invariant to the spatial translation. Particularly for the variance statistic, we can mathematically define the stationary property of a noise $\mathcal{N}$ as below:

[TABLE]

where $\ell$ denotes the translation shifts. To compel the satisfaction of stationary property, we first partition the predicted noise into $B$ non-overlapping $b\times b$ blocks $\{\mathcal{B}_{i}\}_{i=1}^{B}$ followed by computing the standard deviation $\hat{\sigma}_{b}$ within each block $\mathcal{B}_{i}$, resulting in a set $\mathcal{V}=\{\hat{\sigma}_{b}\in\mathbb{R}\}_{b=1}^{B}$. Being a stationary noise implies all elements of $\mathcal{V}$ to be the identical, i.e.:

[TABLE]

Therefore, we apply a Softmax on the elements in $\mathcal{V}$ to get a probability distribution $\Uppsi=\{\psi_{i}\in[0,1]\}$ over all blocks. Ideally, $\Uppsi$ should give the same probability estimation for every block. To measure this, we compute $\mathcal{L}_{st}$ by computing the cross-entropy between $\Uppsi$ and a discrete uniform distribution over $B$ blocks. In our implementation, we randomly select $b$ from $\{2,4,8,16\}$ for each update.

Piecewise Constant Loss. Based on the prior suggested by Mumford Shah [21], $\mathcal{L}_{pc}$ is designed to encourage our model to output signal estimates that contain constant intensity values for all pixels within very small segments. To compute $\mathcal{L}_{pc}$, we first need to simulate the piecewise constant counterpart of the noisy input. To do so, a graph-based segmentation [10] method is firstly used to partition the noisy image $\mathcal{N}$ into $K$ pixel clusters $\{\mathcal{C}_{k}\}_{k=1}^{K}$, where $\mathcal{C}_{k}$ refers to the set of indices of pixels that belong to the $k$th cluster. We subsequently replace the values of pixels within each cluster with their average intensities, resulting in $\mathcal{M}=\{m_{i}\in\mathbb{R}\}_{i=1}^{M}$. Afterward, we measure $\mathcal{L}_{pc}$ by computing the $L_{2}$ distance between the gradients of the signal estimate $\mathcal{S}$ and simulated piecewise constant image $\mathcal{M}$:

[TABLE]

where $\mathcal{G}_{x}(\cdot)$ and $\mathcal{G}_{y}(\cdot)$ denote the gradient operation across horizontal and vertical axes. The reason for comparing the gradients is obvious as we do not desire to encourage the network to produce pixel intensities of $\mathcal{M}$, but the smoothness property of the pixels within each cluster.

Total Variation Loss. $\mathcal{L}_{tv}$ regularizes the model to preserve large-scale edges and textures of the image while smoothing out the noise gradients. We use the standard formulation of total variation [23] in our implementation. Given the signal estimate $\mathcal{S}$, $\mathcal{L}_{tv}$ can be written as:

[TABLE]

4 Experiments

Implementation Details. Noisy images are obtained by synthetic contamination of the clean images with additive white Gaussian noise. During the training, we randomly crop images into patches of size $64\times 64$ and augment them by random 90*∘* rotations and horizontal/vertical flip, however, the quantitative evaluation is reported on the full-size images of the test set. All networks are trained for 500 epochs with a batch size of 16 using Adam optimizer with default parameters. The initial learning rate is set to $10^{-4}$ and is halved every 100 epochs. For Noise2Noise and Noise2Self, we replicate the original training settings reported by the authors. We use publicly released codes to implement non-local mean (NLM) and BM3D denoising methods. It is noteworthy that all loss terms in the loss function have equal coefficients set to 1, except for the total variation term which is set to $5\times 10^{-5}$.