Deep Single Image Deraining Via Estimating Transmission and Atmospheric Light in rainy Scenes

TL;DR

This paper introduces a novel deep learning approach for single image deraining that estimates transmission and atmospheric light using an atmospheric scattering model, leading to improved rain removal performance.

Contribution

It proposes a triangle-shaped network for estimating atmospheric light and integrates it with transmission estimation based on an atmospheric scattering model for better deraining results.

Findings

Outperforms state-of-the-art deraining methods in subjective and objective evaluations.

Utilizes a robust method for estimating atmospheric light in rainy scenes.

Employs ShuffleNet Units for efficient transmission map learning.

Abstract

Rain removal in images/videos is still an important task in computer vision field and attracting attentions of more and more people. Traditional methods always utilize some incomplete priors or filters (e.g. guided filter) to remove rain effect. Deep learning gives more probabilities to better solve this task. However, they remove rain either by evaluating background from rainy image directly or learning a rain residual first then subtracting the residual to obtain a clear background. No other models are used in deep learning based de-raining methods to remove rain and obtain other information about rainy scenes. In this paper, we utilize an extensively-used image degradation model which is derived from atmospheric scattering principles to model the formation of rainy images and try to learn the transmission, atmospheric light in rainy scenes and remove rain further. To reach this goal, we propose a robust evaluation method of global atmospheric light in a rainy scene. Instead of using the estimated atmospheric light directly to learn a network to calculate transmission, we utilize it as ground truth and design a simple but novel triangle-shaped network structure to learn atmospheric light for every rainy image, then fine-tune the network to obtain a better estimation of atmospheric light during the training of transmission network. Furthermore, more efficient ShuffleNet Units are utilized in transmission network to learn transmission map and the de-raining image is then obtained by the image degradation model. By subjective and objective comparisons, our method outperforms the selected state-of-the-art works.

Tables

Table 1. Table 1: PSNR and SSIM comparisons of selected state-of-the-art and our methods on our two datasets.

Baseline	Rain-I		Rain-II
Metric	PSNR	SSIM	PSNR	SSIM
[12]	29.10	0.873	30.01	0.895
[25]	27.51	0.897	26.68	0.830
[38]	26.66	0.885	25.33	0.867
[37]	27.69	0.851	29.97	0.893
Ours	31.35	0.898	34.41	0.936

Table 2. Table 2: Average complexity comparisons of selected methods and our methods on our two datasets. The image size is 512 × 512 512 512 512\times 512 .

Methods	[12] (CPU)	[25] (GPU)	[38] (GPU)	[37] (GPU)	Ours (GPU)
Time	$4.06s$	$0.47s$	$0.06s$	$1.39s$	$0.03s$

Table 3. Table 3: PSNR and SSIM of our ablation studies.

Variants	Rain-I		Rain-II
Metric	PSNR	SSIM	PSNR	SSIM
DEMO-NetA1	27.15	0.772	25.48	0.793
DEMO-NetA2	27.49	0.806	28.57	0.844
DEMO-NetA3	28.13	0.843	29.07	0.874
DEMO-Net	31.35	0.898	34.41	0.936

Equations

$$\mathbf{I}(\mathbf{x})=\mathbf{T}(\mathbf{x})\mathbf{J}(\mathbf{x})+(1-\mathbf{T}(\mathbf{x}))\mathbf{A},$$

$$\mathbf{T}(\mathbf{x})=e^{-{\int }_0^d\beta (\mathbf{x},s)ds},$$

$$\tilde{\mathbf{T}}(\mathbf{x})=1-mi{n}_c(mi{n}_{\mathbf{y}\in \Omega (\mathbf{x})}(\frac{I^c(\mathbf{y})}{A^c})).$$

$$J^{dark}(\mathbf{x})=mi{n}_{c\in \{r,g,b\}}(mi{n}_{\mathbf{y}\in \Omega (\mathbf{x})}(J^c(\mathbf{y})))$$

$$\mathbf{J},\mathbf{A},\mathbf{T}=\mathcal{D}(\mathbf{I}),$$

$$\hat{\mathbf{J}}=(\mathbf{I}-(1-\mathbf{T})\circ \mathbf{A})\oslash \mathbf{T},$$

$$\hat{\mathbf{A}}=\mathcal{A}(\mathbf{I}),$$

$$\hat{\mathbf{T}}=\mathcal{T}(\mathbf{I}),$$

$$\hat{\mathbf{J}}=(\mathbf{I}-(1-\mathcal{T}(\mathbf{I}))\circ \mathcal{A}(\mathbf{I}))\oslash \mathcal{T}(\mathbf{I}).$$

$${\mathcal{L}}_{\mathbf{A}}=\sum _{t=1}^N\Vert \mathcal{A}({\mathbf{I}}_t)-{\tilde{\mathbf{A}}}_t{\Vert }_F^2,$$

$$\mathcal{L}=\sum _{t=1}^N\Vert \mathcal{D}({\mathbf{I}}_t)-{\mathbf{J}}_t{\Vert }_F^2,$$

Full text

1 Introduction

Images and videos are becoming ubiquitous for keeping our favorite memories alive. However, the visual quality of outdoor images and videos are highly affected by weather conditions. For instance, rain occludes and distorts parts of the image that might be of great interest for both personal and commercial users. This, consequently, degrades the performances of many computer vision algorithms e.g. [1]. Typically rain and other distortions in images and videos are dealt with by denosing where the objective is to remove additional undesired artifacts in the image. In applications where outdoor images are primarily used (such as driverless cars111See the Bloomberg Businessweek article ‘Self-driving cars can handle neither rain nor sleet nor snow’ on 17 Sept. 2018), urgent attention to denoising is required. In particular, rain as one of the most prominent forms of artifacts in outdoor scenes poses unique set of challenges. The pattern of rain streaks are complex, and some of them even merge into the background and cannot be seen. Some small image features which are utilized by some outdoor vision systems and computer vision algorithms will be destroyed seriously. Besides, due to the scattering of rain to light, the contrast of an image becomes really low, which also influences the performances of some algorithms.

Conventional rain removing methods in videos exploited the correlation information among frames to localize and remove rain streaks in either spatial or frequency domain, e.g. [2, 3, 4, 5]. For single images, however, rain removal is always more challenging as the problem is indeed an ill-posed one [38]. There are no algorithms or models by which we can obtain accurate de-raining results. All the existing methods can only give an evaluation. Dictionary-learning method [28] was first utilized to remove rain in single images [13, 19, 6, 34, 17, 27, 33]. These approaches are computationally demanding when learning an over-complete dictionary. Besides, the feature descriptors that are utilized to identify rain atoms are not accurate, as such often image details are mistreated as rain and vice versa. Gaussian mixture model-based layer priors [26] accommodate multi-scale and multi-orientation rain streaks, but occassionally miss image details and is computationally demanding.

Deep learning has emerged as a powerful tool in achieving state-of-the-art results in many applications and showed to improve de-raining performances greatly. These improvements are not only in visual quality but also in processing speed [12, 11, 37, 38, 22]. These methods either (1) learn a clear image directly from the given rainy one or (2) estimate a rain residual which is subtracted from the observed rainy image to obtain the resulting de-rained one. However physical aspects of the scene such as atmospheric light and transmission are generally ignored. These physical variables or an image degradation model which is derived from robust physical principle describe the influence of atmospheric particles (including rain) on images more completely and accurately. An accurate model also boost the de-raining performances. Besides, existing de-raining methods tend not to obtain good results for rain streaks whose edges are blurry and merge into the background.

To solve the above problems, we utilize an image degradation model which is derived from the principles of scattering medium (e.g. fog, rain and snow) to describe the formation of rainy images. This model reflects the interaction of light and medium and expresses the influence of bad weather on images more accurately [10]. However, this model was always used in the dehazing task but not de-raining. The major issue is that dynamic rain causes random spatial variations which are more hard to evaluate the atmospheric light and transmission than the static haze.

In this paper, we consider using the environment’s physical parameters in a deep learning model for de-raining. In particular, we propose a novel neural architecture that is composed of two components that are jointly trained in an end-to-end fashion: (1) a transmission and (2) atmospheric light estimator. We show the combination of these two aspects learns to remove rain significantly better and achieves state-of-the-art results. In this framework, we first pre-train a triangle-shaped network with initial estimates of the atmospheric light. Subsequently, in joint training with the transmission, we fine-tune it. This approach allows us to handle wide rain streaks with blurred edges for which the existing methods performance is sub-par. In Figure 1, we show one example of rain-removed result from the given rainy image along with the transmission found by our model. In the transmission, rain streaks become more apparent, especially these slim rain streaks which are merged in the background and cannot be seen in the original rainy image as well as the shape, size, and density. The whole map clearly shows the attenuation of rain to the reflected light.

In summary, the contributions of this paper are:

• •

We utilize an image degradation model to describe the formation of rainy images, based on which, we calculate the atmospheric light, transmission, and de-raining result simultaneously in a rainy scene.

• •

We propose a method to evaluate the atmospheric light in a single rainy image, which has been an unsolved issue.

• •

We design a triangle-shaped network which utilizes the global image information to learn the global atmospheric light, and more efficient revised ShuffleNet units are utilized to construct our network to learn transmission in the rainy scenes.

• •

Compared to the state-of-the-art de-raining works, our method obtains better rain-removed images in objective and subjective assessments. We also show the generalization of our model by using it on de-hazing task.

2 Related Work

Conventional Methods Rain effect was first removed in videos by utilizing the correlation information among video frames. In this paper, we focus on single image de-raining task. Before 2010, learning an over-complete dictionary to express an image sparsely had had very good performance [28]. In a over-complete dictionary, every atom contains some specific image contents. An image can be expressed by these atoms sparsely. Fu et al. [13] naturally used a dictionary to decompose a rainy image, so that some dictionary atoms just include image contents, while the other atoms only contain rain information. Then they use the edge direction of rain to classified rain and non-rain atoms. The rain removed results then can be recovered by orthogonal matching pursuit algorithm [29]. Some improved works which also use dictionary learning appeared afterward [19, 6, 17, 27, 34, 33]. These works designed more robust feature descriptors to enhance the accuracy of identifying rain atoms. Different from other methods the work [33] analyze the common characteristics of rain and snow to design some descriptors (SVCC, PDIP) to remove both rain and snow. However, dictionary based methods are always time-consuming when training an over-complete dictionary. Another important shortcoming which influences the de-raining performance is the non-adaptive feature descriptors which tend not to completely express various rain, so that the generalization of these methods is always low.

To avoid time-consuming dictionary learning stage, some filter (e.g., the edge-preserving guided filter [15]) based de-raining works appeared [9, 36]. These works are always simple, so they have high processing speed, but their de-raining effect is really limited. In [7], Chen et al. proposed a low-rank appearance model to capture the spatio-temporally correlated rain streaks. Li et al. [26] proposed priors which are based on Gaussian mixture models for both rain and background to accommodate multiple orientations and scales of the rain streaks. However, these two models will mistreat some image details as rain and remove them with rain streaks together. Besides, the work [26] is more time-consuming than dictionary learning methods.

Deep Learning Based Methods In recent three or four years, deep learning has been widely used in rain removal task and obtain better rain-removing performances compared with conventional methods. However, all these methods either estimate backgrounds directly from rainy images, or learn a rain residual layer which is subtracted from rainy image to obtain the rain-removed result. No other information about rainy scene is acquired.

Different from other common strategies, Fu et al. trained their DerainNet in high-frequency domain instead of image domain to extract image details to improve visual quality [11]. In the meantime, inspired by deep residual network (ResNet) [16], a deep detail network which is also trained in high-pass domain was proposed to reduce the mapping range from input to output, to make the learning process easier [12]. Though training networks on high-frequency details can remove the interference from the background to a degree, some bright rain streaks cannot be filtered completely from low-frequency part so that some rain streaks will remain in the final results for some rainy images. Yang et al. decomposed a rainy image into rain layer and background layer and added a binary map to locate rain streaks. Besides, they created a new model which includes two components to express rain streak accumulation and various shape and directions of rain streaks [37]. With the help of binary map, this method can deal bright rain streaks, but tend to neglect some rain streaks with blurring edges. In [38], Zhang et al. tried to automatically estimate rain density by network itself, then a multi-stream densely connected DID-MDN structure which can better characterize rain streaks with various shape and size is trained to remove rain streaks guided by the estimated rain density. However, this work can cause blur for some images with fine details.

A multi-stage network which consists of several parallel sub-networks was designed to model and remove rain streaks of various size [23]. Different parallel sub-networks model rain streaks with corresponding sizes. Li et al. regarded rain streaks as the accumulation of multiple rain streaks layers, then use a recurrent neural network to remove rain streaks state-wisely. Though, recurrent training method is used, this work is not sensitive to rain streaks with blur edges. In [22], a non-locally enhanced encoder-decoder network framework is proposed to capture long-range spatial dependencies via skip-connections and pooling indices guided decoding is used to learn increasingly abstract feature representation to preserve the image details decoding. Different from all these state-of-the-art de-raining works, we design a simple but effective network to estimate atmospheric light, transmission of rain and obtain a clear rain-removed results. For the wide rain streaks with blurry edges, our method also produces better visual effect.

3 Background

By 2001, Nayar and Narasimhan had made detailed studies to the influence of bad weather (e.g., haze, rain, snow) on images/videos in computer vision by utilizing the scattering and absorption principle of medium to light [31, 30], and several expressions had be made to reflect the degradation of bad weather to images/videos. Global atmospheric light can be approximated as a constant $3\times 1$ vector, then a simplified image degradation model can be obtained [10]:

[TABLE]

where $\mathbf{I(\mathbf{x})}$ is the observed intensity, $\mathbf{J(\mathbf{x})}$ is scene radiance [14], $\mathbf{A}$ is the global atmospheric light and $\mathbf{T}(\mathbf{x})$ is the medium transmission to describe the attenuation of medium to light. $\mathbf{T}(\mathbf{x})\mathbf{J}(\mathbf{x})$ is the attenuated scene intensity (radiance) by the medium called direct attenuation [32]. $(1-\mathbf{T}(\mathbf{x}))\mathbf{A}$ is the scattered light called airlight [20, 32] which leads to the color shift of the scene. The more general expression of transmission is:

[TABLE]

where $\beta(\mathbf{x},s)$ is the scattering coefficient of the medium [14] which is relative to the pixel location $\mathbf{x}$ ($\mathbf{x}$ corresponds to a scene point) and the distance $d$ of scene to the camera.

Haze is homogeneous approximately in a small patch, its scattering coefficient $\beta(\mathbf{x},s)$ can be regarded as a constant value, which gives great convenience to remove haze from single images by Eq. 1 [30, 31, 32, 10]. The dark channel prior [14] helps to better evaluate transmission $\mathbf{T}(\mathbf{x})$ and the global atmospheric light $\mathbf{A}$. However, the assumption that $\beta(\mathbf{x},s)$ is constant is not satisfied any more in rainy scenes. Hence, the following estimation of transmission [14] cannot be obtained:

[TABLE]

Besides, the definition of dark channel:

[TABLE]

is also not significant, because rain streaks are merged into the dark channel, and some even disappear. In the above two equations, $J^{c}$ is a color channel of $\mathbf{J}$, $\Omega(\mathbf{x})$ is a local patch centered at $\mathbf{x}$. In Figure 2, we show the transmission and dark channel of a rainy image calculated by (3)(4) respectively. From Figure 2 (b)(d), we can see that there is not so much rainy information in the dark channel and estimated transmission. Hence, the dark channel prior cannot estimate global atmospheric light $\mathbf{A}$ as like in [14]. Figure 2(e) is the de-raining result by the dark channel prior, which proves further that dark channel prior cannot be used in de-raining task.

Though Eq. (1) is derived to describe the image degradation caused by the scattering of several kinds of atmospheric particles (e.g., haze, rain, snow), no one has used it in de-raining task before. In the next section, we will present a new method to estimate global atmospheric light $\mathbf{A}$ in rainy scenes, design a simple and efficient network to estimate transmission of rain and remove rain by Eq. (1) further.

4 Our Method

Given a rainy image $\mathbf{I}$, the goal of our paper is to recover a rain-free image $\mathbf{J}$, estimate the global atmospheric light $\mathbf{A}$ and calculate rain’s transmission $\mathbf{T}$. Our immediate thought is to build a DEgradation MOdel based Network (DEMO-Net) to estimate $\mathbf{J}$, $\mathbf{A}$ and $\mathbf{T}$ from a single rainy image $\mathbf{I}$:

[TABLE]

where $\mathcal{D}(\cdot)$ denotes the mapping of our DEMO-Net and its whole network structure is shown in Figure 3. Different from many previous deep learning based works which learn $\mathbf{J}$ directly from $\mathbf{I}$ or estimate a rain residual $\mathbf{R}$ ($\mathbf{J}=\mathbf{I}-\mathbf{R}$). We utilize a more accurate and complete model to express rainy images to not only remove rain but also obtain two important physical variables in rainy scenes. Before introducing our network $\mathcal{D}(\cdot)$, we first estimate global atmospheric light $\mathbf{A}$ which is really important for our whole work.

4.1 Estimating the Global Atmospheric Light

Airlight ($(1-\mathbf{T}(x))\mathbf{A}$) is the result of $\mathbf{A}$ scattered by atmospheric particles [20]. Hence, $\mathbf{A}$ can be approximated by the highest intensity of particles in images. Based on the same principle, a pixel with the highest intensity is used to estimate $\mathbf{A}$ for hazy images in [32], [10, 14] refine its estimation. In actual, He et al. [14] utilized the dark channel prior to locate haze (the parts which have relatively high values in dark channel prior), then use the top $0.1\%$ brightest pixels in dark channel to determine $\mathbf{A}$. The influence of rain on images is more complex than haze and the dark channel prior loses function for rainy images.

According to [20], in rainy scenes, the highest value of rain pixels can be used to estimate $\mathbf{A}$. So we must locate rainy pixels first, then find the highest value as the estimation of $\mathbf{A}$. In [37], Yang et al. designed a deep neural network to localise rainy pixels. However, the authors used Softmax to approximate binary training samples. Hence, we use a threshold to obtain a binary location map, then find the rainy pixel with highest value to approximate $\mathbf{A}$. Some location maps are shown in supplement.

4.2 Proposed DEMO-Net

According to the image degradation model in Eq. (1), given a rainy image $\mathbf{I}$, we can obtain its rain-removed image by the following expression if the atmospheric light $\mathbf{A}$ and the transmission $\mathbf{T}$ can be known first:

[TABLE]

where $\widehat{\mathbf{J}}$ is the estimation of $\mathbf{J}$, $\circ$ and $\oslash$ denote pixel-wise multiplication and division respectively.

In Section 4.1, we have obtained an initial evaluation for $\mathbf{A}$. In order to acquire an estimation of $\mathbf{A}$ which is suit better to our DEMO-Net to obtain a clear rain-removed result and good transmission, we train a triangle-shaped neural network to learn atmospheric light for every rainy image which is supervised by our estimated $\mathbf{A}$ in Section 4.1. We use $\mathcal{A}(\cdot)$ to denote the mapping of this network. Training a network can synthesize the information from many scenes and obtain more robust estimation, which is equal to learning a general distribution of atmospheric light $\mathbf{A}$.

After obtaining $\mathbf{A}$, we utilize a novel network denoted as $\mathcal{T}(\cdot)$ to calculate the transmission $\mathbf{T}$. $\mathcal{T}(\cdot)$ and $\mathcal{A}(\cdot)$ are composed based on the model (1). By Eq. (6), we finally calculate the rain-removed result (shown in Figure 3). During the training for transmission, $\mathcal{T(\cdot)}$ and pre-trained $\mathcal{A(\cdot)}$ estimate $\mathbf{T}$ and $\mathbf{A}$ respectively, then Eq. (6) is used to calculate rain-removed result in the forward propagation. Eq. (6) is differentiable to $\mathbf{A}$ and $\mathbf{T}$, hence in the back propagation, we use the difference between the ground-truth $\mathbf{J}$ and calculated de-raining result $\widehat{\mathbf{J}}$ to not only update the parameters of $\mathcal{T(\cdot)}$, but also fine-tune the parameters of $\mathcal{A(\cdot)}$ with smaller learning rate to boost the performances. This is also one of the reasons why we use network to re-estimate $\mathbf{A}$ rather than using initially-estimated $\mathbf{A}$ directly.

4.3 Network Structure for $\mathcal{A(\cdot)}$

$\mathcal{A(\cdot)}$ takes the rainy image $\mathbf{I}$ as input to estimate $\mathbf{A}$:

[TABLE]

where $\widehat{\mathbf{A}}$ is the estimation of $\mathbf{A}$. According to [20, 30, 31], atmospheric light is a $3\times 1$ vector. Hence, we design layer-wise down-sampling triangle-shaped convolution network, shown in Figure 3, so that we can obtain a $3\times 1$ vector. A convolution layer is first used to extract features primarily. Then five convolution layers with stride equal to $2$ are utilized to extract information and down-sample the extracted feature map. Meanwhile, the number of channels also doubles successively with the down-sampling. Every convolution layer is followed by batch normalization [18] and rectified linear unit [21]. When an image is large, we cannot obtain a $3\times 1$ vector just by $5$ convolution layers. In order to adapt more images with different sizes, an adaptive average pooling is utilized after $5$ down-sampling convolution layers to down-sample the features to $1\times 1$ size directly. At last, a fully connected layer synthesizes the whole extracted information into a $3\times 1$ vector, namely our global atmospheric light $\mathbf{A}$.

4.4 Network Structure for $\mathcal{T(\cdot)}$

Before introducing our network structure for $\mathcal{T(\cdot)}$, we first pay attention to the basic unit structures. In [39], Zhang et al. proposed an extremely computation-efficient and performance-preserving ShuffleNet structure. In ShuffleNet units, pointwise group convolution and channel shuffle are used to boost the propagation of information flow. Group convolution helps to encode more information and channel shuffling enables different groups to share cross-group information. The reason why we use ShuffleNet units is that the influence of rain is complex, its sizes and patterns are various. ShuffleNet units make best use of the extracted features to estimate a better transmission.

In order to adapt to the complex rain, revised ShuffleNet units are used in our work, shown in Figure 4. We deepen the extracted features of single unit by adding another $3\times 3$ depthwise convolution [8]. To reduce the influence of padded [math] in common convolution, we use symmetrically padded depthwise convolution (SDWConv). Pointwise group convolution and channel shuffling also double to boost information pass.

$\mathcal{T}(\cdot)$ also takes the rainy image $\mathbf{I}$ as input to estimate the transmission $\mathbf{T}$:

[TABLE]

where $\widehat{\mathbf{T}}$ is the estimation of $\mathbf{T}$. In the network structure, shown in Figure 3, we first use a convolution layer to extract feature shallowly. Then two ShuffleNet units which use concatenation method to combine the bottleneck and the shortcut (ShuffleUnit(cat)) are adopted to down-sample the feature map and extract feature. Furthermore, $6$ ShuffleNet units which use addition method to combine the bottleneck and the shortcut (ShuffleUnit(add)) are utilized to deepen our network. Please refer to [39] for the details of ShuffleNet unit. At last, we up-sample the feature map to original size and convolution layers are followed to fuse multi-channel features. By a Sigmoid activation function, the transmission can be obtained.

Based on the above definition, the rain-removed result $\widehat{\mathbf{J}}$ can be obtained by:

[TABLE]

4.5 Training Loss

Our DEMO-Net is trained by two steps. If we utilize $\widetilde{\mathbf{A}}$ to denote the global atmospheric light estimated initially in Section 4.1, we first train $\mathcal{A}(\cdot)$ on the given training samples $\{(\mathbf{I}_{t},\widetilde{\mathbf{A}}_{t})\}^{N}_{t=1}$. Then $\mathcal{T}(\cdot)$ is trained on the training samples $\{(\mathbf{I}_{t},\mathbf{J}_{t})\}^{N}_{t=1}$ ($\mathbf{J}_{t}$ is the ground-truth of $\mathbf{I}_{t}$) and $\mathcal{A}(\cdot)$ is fine-tuned during this training process.

The two-step trainings both utilize MSE loss function:

[TABLE]

[TABLE]

where $\mathcal{L}_{\mathbf{A}}$ and $\mathcal{L}$ are losses for $\mathcal{A}(\cdot)$ and $\mathcal{T}(\cdot)$ respectively. Some training details are shown in supplement.

5 Experiments

In order to assess the performance of our rain-removing method quantitatively, the commonly used PSNR and SSIM [35] are used as our metrics. For visual quality evaluation, we give the rain-removed results of some real-world and synthetic rainy images. Because the authors of [22] do not want to release their code so far, we utilize another four state-of-the-art works [37, 38, 12, 25] to make comparisons.

5.1 Atmospheric Light and Transmission

Before comparing with selected works, we show the atmospheric light and transmission estimated by our method first. Figure 5 shows the results of two real-world rainy images. The atmospheric light estimated by the method in Section 4.1 tend to change relatively apparently for different rainy images, while the values estimated by $\mathcal{A}(\cdot)$ are much more stable. According to Eq. (1)(2), when $d\to\infty$, $\mathbf{T(\mathbf{x})}=0$ and $\mathbf{I(\mathbf{x})}=\mathbf{A}$. Hence, $\mathbf{A}$ is the intensity of light before being scattered by medium, it should be a relatively stable and large value, hence our network $\mathcal{A}(\cdot)$ gives a better estimation to $\mathbf{A}$. We will also show later that the rain-removed results of using $\mathcal{A}(\cdot)$ are much better.

Our method obtains good de-raining results (Figure 5(b)), and transmission gives us more information of rain. First, the shapes of rain streaks are more apparent, especially, some rain streaks merged in the background. Second, the influence of rain on background is also clearer, the pixels having high values attenuate much more light. Some other information, such as rain density and accumulation are also seen more apparently.

5.2 Datasets

In our work, we randomly select $6000$ training samples from the dataset in [38] and [24] respectively, so our training dataset includes $12000$ samples. One of our testing datasets Rain-I (has $400$ samples) is composed by randomly selecting $100$ samples from the testing datasets of [38, 25, 12, 37] respectively, which are selected works we to make comparisons. As the synthetic rainy images in [24] are different from others, our another testing dataset Rain-II ($400$ samples) is randomly selected from the testing dataset in this work. The real-world rainy images come from the previous works, and some of them are from Google.

5.3 Quantitative Evaluation on Synthetic Datasets

We show the PSNR/SSIM comparisons with the selected state-of-the-art rain-removing works in Table 1. We can see that our method obtains higher PNSR/SSIM on both datasets. The PSNR is about $4$ dB higher than the best state-of-the-art [12] on Rain-II. In this dataset, the synthetic rain streaks are wide and their edges are not clear and emerge into the background. Figure 6 are two synthetic rainy images that state-of-the-art cannot handle well. By comparison, our method produces better rain-removed results, nearly all rain streaks disappear. For the second one, though we remove the heavy rain streaks, some traces still remain in our results. More results are in supplement.

5.4 Qualitative Evaluation on Real-world Images

In Figure 7, we show some results of selected and proposed methods on real-world images. When encountered with wide rain streaks, selected methods cannot produce desired results, some apparent rain streaks remain in the resulting images (e.g., the first one). For narrow rain streaks (e.g., third one), all the methods can acquire comparable results. For the rain streaks which have blur edges (e.g. the second one), the method [38] and our method obtain better results. The complexity comparisons are shown in Table 2, we can see our work is the fastest one.

5.5 Ablation Studies

To verify the performances of our whole network further, we do some ablation experiments. Our network has three different ablation variants. One is removing $\mathcal{A}(\cdot)$, and using atmospheric light estimated by the method in Section 4.1 to train the transmission network $\mathcal{T}(\cdot)$, we call it DEMO-NetA1 to simplify our expression. The second one is that $\mathcal{A}(\cdot)$ and $\mathcal{T}(\cdot)$ are trained jointly, but the pre-trained parameters for $\mathcal{A}(\cdot)$ is not utilized, which is called DEMO-NetA2. In the last one, we use pre-trained $\mathcal{A}(\cdot)$ to estimate atmospheric light, but its parameters will not be updated during the training for $\mathcal{T}(\cdot)$, we call it DEMO-NetA3. In Figure 8 we show the ablation study results. The objective indexes are shown in Table 3. We can see that DEMO-NetA1 has lowest performance, which may be related to the relatively large variance of estimated $\mathbf{A}$ for different images by the method in Section 4.1. This situation has good convergence during the training, but its generalization is low. With the introduce of $\mathcal{A}(\cdot)$, the performance is enhanced. DEMO-NetA3 is better than DEMO-NetA2, which proves that our estimated $\mathbf{A}$ in Section 4.1 is useful. Our DEMO-Net possesses the apparent best performances, we use the pre-trained parameters for $\mathcal{A}(\cdot)$, then fine-tuned them during the training of $\mathcal{T}(\cdot)$, which is equal to adding a prior of the global atmospheric light to the training of the whole network.

5.6 Other Potentials of Our Network

Our work still has some potentials to handle images obtained under other weather conditions. In Figure 9 we simply show some dehazing results by our DEMO-Net. Besides, if the initial $\mathbf{A}$ for snow can be obtained, maybe our method can also work on snowy images.

6 Conclusions

In this paper, we utilize an image degradation model which is derived from atmospheric scattering principle to model the formation of rainy images. In order to remove rain effect and obtain the transmission which contain more information of rain, we proposed a method based on the scattering principle to estimate the global atmospheric light initially. To estimate a better global atmospheric light and enhance the de-raining performance, we designed a special triangle-shaped network to learn global atmospheric light for rainy images supervised by the initial estimation. Then a simple but novel network which boots information pass was used to learn the transmission, during this process, the triangle-shaped network is fine-tuned further. Results on synthetic and real-world rainy images show that our method outperforms the selected state-of-the-art, and obtains a transmission which contain more information in rainy scene.