Pyramid Real Image Denoising Network

TL;DR

This paper introduces PRIDNet, a pyramid-based neural network for real-world image denoising that effectively models complex noise through multi-stage processing, achieving state-of-the-art results.

Contribution

The paper proposes a novel multi-stage pyramid denoising network with channel attention and adaptive feature fusion for improved real-world noise removal.

Findings

Achieves competitive results on real noisy datasets.

Outperforms existing denoisers in both quantitative and visual quality.

Utilizes multi-scale features and adaptive fusion for better noise modeling.

Abstract

While deep Convolutional Neural Networks (CNNs) have shown extraordinary capability of modelling specific noise and denoising, they still perform poorly on real-world noisy images. The main reason is that the real-world noise is more sophisticated and diverse. To tackle the issue of blind denoising, in this paper, we propose a novel pyramid real image denoising network (PRIDNet), which contains three stages. First, the noise estimation stage uses channel attention mechanism to recalibrate the channel importance of input noise. Second, at the multi-scale denoising stage, pyramid pooling is utilized to extract multi-scale features. Third, the stage of feature fusion adopts a kernel selecting operation to adaptively fuse multi-scale features. Experiments on two datasets of real noisy photographs demonstrate that our approach can achieve competitive performance in comparison with state-of-the-art denoisers in terms of both quantitative measure and visual perception quality. Code is available at https://github.com/491506870/PRIDNet.

Tables

Table 1. TABLE I: Quantitative performance of our model on DND compared with other published techniques, and sorted by sRGB PSNR.

	Raw		sRGB
Algorithm	PSNR	SSIM	PSNR	SSIM	Blind / Non-blind
TNRD [17]	44.97	0.9624	33.65	0.8306	Non-blind
BM3D [5]	46.64	0.9724	34.51	0.8507	Non-blind
KSVD [18]	45.54	0.9676	36.49	0.8978	Non-blind
WNNM [19]	46.30	0.9707	37.56	0.9313	Non-blind
FFDNet [6]	-	-	37.61	0.9415	Non-blind
DnCNN [2]	-	-	37.90	0.9430	Blind
CBDNet [7]	-	-	38.06	0.9421	Blind
Path-Restore [8]	-	-	39.00	0.9542	Blind
N3Net [3]	47.56	0.9767	-	-	Blind
PRIDNet	48.48	0.9806	39.42	0.9528	Blind

Table 2. TABLE II: Ablation study of our model on validation dataset of SIDD.

Channel Attention	Pyramid	Kernel Selecting	SIDD
-	-	-	51.81
-	$\surd$	$\surd$	52.02
$\surd$	-	$\surd$	51.98
$\surd$	$\surd$	-	52.15
$\surd$	$\surd$	$\surd$	52.20

Equations

$$\mu =Sigmoid(F{C}_2(ReLU(F{C}_1(GAP(U))))).$$

$$U^{\prime }=U\circ \mu ,$$

$$\overline{U}=U^{\prime }+U^{\prime \prime }+U^{\prime \prime \prime }.$$

$$k_c=\frac{e^{k_c^{\prime }}}{e^{{\alpha }_c^{\prime }}+e^{{\beta }_c^{\prime }}+e^{{\gamma }_c^{\prime }}},k=\alpha ,\beta ,\gamma ,$$

$$V_c={\alpha }_c\cdot U^{\prime }+{\beta }_c\cdot U^{\prime \prime }+{\gamma }_c\cdot U^{\prime \prime \prime },$$

Full text

I Introduction

Image denoising aims at restoring a clean image from its noisy one, which plays an essential role in low-level visual tasks. There has been considerable research on it, and near-optimal performances have been achieved for the removal of statistical distribution-regular noise (e.g., additive white Gaussian noise (AWGN), shot noise). Nevertheless, there is still a huge difference between specific noise and real-world noise. Among the latter, the noise comes from both shooting environment and image processing pipeline, thus its forms demonstrate complexity and diversity.

Recently deep Convolutional Neural Networks (CNNs) have led to significant improvements on denoising for specific noise. Mao et al. [1] present a very deep fully convolutional encoding-decoding framework with symmetric skip connections for Gaussian denoising, termed REDNet. Zhang et al. [2] demonstrate that by merging residual learning and batch normalization, a denoising CNN (DnCNN) could outperform traditional non-CNN based methods. Other CNN methods [3, 4] also obtain promising denoising performance.

However, once the methods targeting for the specific noise above generalize to real-world noise, the performance may be even worse than the representative traditional methods such as BM3D [5]. Few blind denoising approaches especially for real noisy images are developed. By interactively setting relatively higher noise level, FFDNet [6] can deal with more complex noise. CBDNet [7] further utilizes a noise estimation subnetwork, so that the entire network could achieve end-to-end blind denoising.

Yu et al. [8] propose a multi-path CNN named Path-Restore, which could dynamically select an appropriate route for each image region, especially for the varied noise distribution of a real noisy image. As a commercial plug-in for Photoshop, Neat Image (NI) is also for bind denoising.

Despite these methods have significantly improved the performance of real image denoising, there still remains three issues to be noticed:

First, in most of CNN based denoising methods, all channel-wise features are treated equally without adjustment according to their importance. In CNNs, different feature channels capture different types of noise across all regions of a single noisy image. Among them, some noise are more significant than others, thus should be assigned more weights.

Second, the previously mentioned methods, with fixed receptive fields, fail to carry diverse information. Referring to the traditional denoising method BM3D [5], it searches for similar blocks in the whole image, taking the global information into consideration. Since features are not limited to a small area, receptive fields with different scales can fully exploit hierarchical spatial features. Context information would be very helpful when the image suffers from heavy noise.

Third, for the aggregation of multi-scale features, most of existing methods simply combine them in an element-wise summation manner or just concatenate them together [1]. Although containing the information of all scales, they treat features with different scales indiscriminately, ignoring the spatial and channel specificity of scale-wise features. Therefore multi-scale features can not be adaptively expressed.

To address these issues, we propose a pyramid real image denoising network (PRIDNet) as shown in Fig. 2. The main contribution of this work is three-fold:

• •

Channel attention: Channel attention mechanism is utilized on the extracted noise features, adaptively recalibrating the channel importance.

• •

Multi-scale feature extraction: We design a pyramid denoising structure, in which each branch pays attention to one-scale features. Benefitted from it, we can extract global information and retain local details simultaneously, thereby making preparation for following comprehensive denoising.

• •

Feature self-adaptive fusion: Within concatenated multi-scale features, each channel represents one-scale features. We introduce a kernel selecting module. Multiple branches with different convolutional kernel sizes are fused by linear combination, allowing different feature maps to express by size-different kernels.

II proposed method

In this section, we will formulate the proposed PRIDNet, including network architecture and three stages.

II-A Network Architecture

As shown in Fig. 2, our model includes three stages: noise estimation stage, multi-scale denoising stage and feature fusion stage. The input noisy image is processed by three stages sequentially. Since all the operations are spatially invariant, it is robust enough to handle input images of arbitrary size. To avoid loss of information, the output of the first stage is concatenated with its input before fed into next stage, likewise the second stage.

II-B Noise Estimation Stage

This stage focuses on extracting discriminative features from input noisy images, which is regarded as an estimation of the noise level [7]. We adopt a plain five-layer fully convolutional subnetwork without pooling and batch normalization, ReLU is deployed after each convolution. In each convolution layer, the number of feature channels is set to 32 (except the last layer is 1 or 3), and the filter size is 3$\times$3.

Before the last layer of stage, a channel attention module [9] is inserted to explicitly calibrate the interdependencies between feature channels. As shown in Fig. 3, the collection of channel weight $\mu=[\mu_{1},\mu_{2},...,\mu_{c}]\in\mathbb{R}^{1\times 1\times{C}}$ is our goal, which is applied to rescale input feature maps $U\in\mathbb{R}^{H\times{W}\times{C}}$ to generate recalibrated features. We first squeeze global information of $U$ into a channel descriptor $\nu\in\mathbb{R}^{1\times 1\times{C}}$ by using global average pooling (GAP). Then, it is followed by two fully connected layers (FC), and the number of channels in middle layer is set to 2. Above process can be formulated as

[TABLE]

The final output of channel attention module (denoted as $U^{\prime}\in\mathbb{R}^{H\times{W}\times{C}}$) is obtained by

[TABLE]

where $\circ$ refers to channel-wise multiplication between $U_{i}\in\mathbb{R}^{H\times{W}}$ and scalar calibration weight $\mu_{i}$, $i=1,2,...C$.

II-C Multi-scale Denoising Stage

The idea of pyramid pooling is widely used in the fields of scene parsing [10], image compression and so on. While to the best of our knowledge, it has never been used in the image denoising. Zhou et al. [11] show that the empirical receptive field of CNN is much smaller than the theoretical one especially on high-level layers, which means that global information is not fully integrated when extracting features. On the contrary, for the removal of noise covering entire image, it has great help to match goal blocks with similar content in the whole image.

To mitigate this problem, we develop a five-level pyramid. Through five parallel ways, the input feature maps are downsampled to different sizes, helping branches gain relatively scale-different receptive fields to capture original, local and global information simultaneously. Pooling kernels are set to 1$\times$1, 2$\times$2, 4$\times$4, 8$\times$8 and 16$\times$16, respectively. Each pooled feature is then followed by U-Net [12], the network composed of deep encoding-decoding and skip connections, for studies have shown that successive upsampling and downsampling are helpful for denoising tasks. Note that five U-Nets do not share weights. At final of this stage, multi-level denoised features are upsampled by bilinear interpolation to the same size, and then concatenated together.

II-D Feature Fusion Stage

In order to choose size-different kernel for each channel within concatenated multi-scale results, inspired by [13], a kernel selecting module is introduced. Details of a kernel selecting module is shown in Fig. 4. The given feature maps $U\in\mathbb{R}^{H\times{W}\times{C}}$ are conducted by three parallel convolutions with kernel size 3, 5 and 7, respectively, to get $U^{\prime}\in\mathbb{R}^{H\times{W}\times{C}}$, $U^{\prime\prime}\in\mathbb{R}^{H\times{W}\times{C}}$ and $U^{\prime\prime\prime}\in\mathbb{R}^{H\times{W}\times{C}}$. We first integrate information from all branches via element-wise summation:

[TABLE]

Then $\overline{U}$ is shrunk and then expanded by passing through a GAP and two FCs, the same operations as in the channel attention module, but no Sigmoid at last. The three outputs of $FC_{2}$, $\alpha^{\prime}\in\mathbb{R}^{1\times 1\times{C}}$, $\beta^{\prime}\in\mathbb{R}^{1\times 1\times{C}}$ and $\gamma^{\prime}\in\mathbb{R}^{1\times 1\times{C}}$ are operated by Softmax, which is applied across branches on the channel-wise, like gating mechanism:

[TABLE]

where $\alpha$, $\beta$ and $\gamma$ denote the soft attention vector for $U^{\prime}$, $U^{\prime\prime}$ and $U^{\prime\prime\prime}$, respectively. Note that $\alpha_{c}$ is the c-th element of $\alpha$, likewise $\beta_{c}$ and $\gamma_{c}$. The final output feature maps $V$ are computed via combining various kernels with their attention weights:

[TABLE]

where $\alpha$, $\beta$ and $\gamma$ need to satisfy $\alpha_{c}+\beta_{c}+\gamma_{c}=1$, and $V=[V_{1},V_{2},...,V_{c}]$, $V_{c}\in\mathbb{R}^{H\times{W}}$. Finally, we utilize a ${1\times 1}$ convolutional layer to compress the dimension to 1 or 3 for feature fusion.

III experiments

III-A Datasets

For training, we utilize 320 image pairs (noisy and clean) both in raw-RGB space and sRGB space from Smartphone Image Denoising Dataset (SIDD) [14]. And we set another 1280 $256\times 256$ crops of 40 images in SIDD as our validation data to conduct our ablation study.

For testing, we adopt two widely used benchmark datasets: DND [15] and NC12 [16]. DND, a benchmark of 50 real high-resolution images, captured by consumer grade cameras. Since only noisy images are provided to the public, PSNR/SSIM of denoised results are obtained through the online submission system111https://noise.visinf.tu-darmstadt.de/. NC12 contains 12 noisy images, we only show the denoising results for qualitative evaluation as the ground-truth clean images are unavailable.

III-B Implementation Details

We train two models, one targeting raw images and the other targeting sRGB images. The whole network is optimized with $L_{1}$ loss, and trained by Adam with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and $\epsilon=10^{-8}$. All models are trained with 4000 epochs, where the learning rate for the first 1500 epochs is $10^{-4}$, and then $10^{-5}$ to finetune the model. We set patch size to $256\times 256$, and batch size is set to 2 for “raw” model while 8 for “sRGB” model. All the experiments are implemented with TensorFlow on an NVIDIA GTX 1080ti GPU.

III-C Comparison with State-of-the-arts

Qualitative and Quantitative Results on DND Benchmark. Following the test protocol and tools provided by the website, we process 1000 bounding boxes in 50 real noisy images. The performance of our models with respect to prior work is shown in Table  I. Note that we do not take unpublished or anonymous work into consideration although they have released results. We can see that although state-of-the-art non-blind Gaussian methods, e.g., TNRD [17], BM3D [5] and WNNM [19], are provided by noise level, they still have poorly performance, mainly due to the much difference between AWGN and real noise. CBDNet [7] and Path-Restore [8] are especially trained for blind denoising of real images, thus yield promising results. Our PRIDNet achieves large PSNR gains (i.e., $\sim 0.9$dB for raw, and $\sim 0.4$dB for sRGB) over the second best method. As for running time, PRIDNet takes about 0.05s to process an $512\times 512$ image.

The visual denoising results by the various methods are shown in Fig. 1 and the supplementary material. CBDNet [7] still contains some noise. DnCNN [2] suffers from edge distortion. BM3D, N3Net and Path-Restore introduce some color artifacts to the denoised results, while FFDNet [6] suffers from the problems of over-smoothing and loss of details and structures. Compared with these state-of-the-arts, our method (PRIDNet) achieves a better denoising performance by removing almost all the noise while preserving more fine textural details of whole image.

Qualitative Comparisons on NC12. The methods we consider for comparison include blind denoising approaches (NC [16], NI and CBDNet [7]), blind Gaussian denoising approaches (DnCNN [2]), and non-blind Gaussian denoising approaches (BM3D [5] and FFDNet [6]). For non-blind methods, we exploit NI to estimate the noise level std. so that they can perform the best visual quality. Due to limited space, we leave the visual comparisons of above methods in the supplementary material.

III-D Ablation Studies

We conduct four ablation experiments to assess the importance of three key components in our PRIDNet. All ablation experiments are evaluated on the validation data of SIDD [14]. We can conclude from Table  II that the design of pyramid feature processing is the most crucial for our real image denoising task, which improves 0.22dB. The combination of three key components effectively boost network performance by 0.39dB compared with the plain network.

IV Conclusion

In this paper, a PRIDNet is presented for blind denoising of real images. The proposed network includes three sequential stages. The first stage explores relative importance of feature channels. At the second stage, pyramid pooling is developed to denoise multi-scale features. At the last stage, the operation of self-adaptive kernel selecting is introduced for feature fusion. Both qualitative and quantitative experiments show that our method achieves competitive performance.

Acknowledgments

This work is supported by National Natural Science Foundation of China (No.61671077, 61671264) and the Fundamental Research Funds for the Central Universities 2019PTB-011.