Light Source Point Cluster Selection Based Atmosphere Light Estimation

TL;DR

This paper introduces a clustering-based method for estimating atmosphere light in defogging algorithms, improving accuracy in light source detection and resulting in better image quality.

Contribution

It proposes a novel clustering statistics approach to accurately estimate atmosphere light value and location, enhancing defogging performance.

Findings

More accurate atmosphere brightness estimation

Improved defogging visual quality

Enhanced objective image quality metrics

Abstract

Atmosphere light value is a highly critical parameter in defogging algorithms that are based on an atmosphere scattering model. Any error in atmosphere light value will produce a direct impact on the accuracy of scattering computation and thus bring chromatic distortion to restored images. To address this problem, this paper propose a method that relies on clustering statistics to estimate atmosphere light value. It starts by selecting in the original image some potential atmosphere light source points, which are grouped into point clusters by means of clustering technique. From these clusters, a number of clusters containing candidate atmosphere light source points are selected, the points are then analyzed statistically, and the cluster containing the most candidate points is used for estimating atmosphere light value. The mean brightness vector of the candidate atmosphere light points in the chosen point cluster is taken as the estimate of atmosphere light value, while their geometric center in the image is accepted as the location of atmosphere light. Experimental results suggest that this statistics clustering method produces more accurate atmosphere brightness vectors and light source locations. This accuracy translates to, from a subjective perspective, more natural defogging effect on the one hand and to the improvement in various objective image quality indicators on the other hand.

Figures

Tables

Table 1. Table 1: Comparison of contrast enhancement ratios and EME indicator.

Image	Contrast enhancement ratios			EME
	Original algorithm	Proposed algorithm		Original Image	Original algorithm	Proposed algorithm
Forest	0.46	0.55		506.08	5207.98	5446.66
Cones	0.94	0.98		0.98	999.58	1129.13
Pumpkins	0.73	0.81		53.36	1611.85	2020.85
Hongkong	0.65	0.96		21.99	5926.53	6365.53
Train	0.98	2.85		0.47	2267.75	2798.29
Swan	2.20	2.36		25.22	2639.50	2444.04

Table 2. Table 2: Blind indicator.

Image	Original algorithm				Proposed algorithm
	$e$	$r$	$\sigma$		$e$	$r$	$\sigma$
Forest	0.111	1.198	0.001		0.111	1.229	0.001
Cones	0.324	1.528	0.000		0.326	1.547	0.000
Pumpkins	0.481	1.715	0.000		0.463	1.759	0.000
Hongkong	0.051	1.362	0.006		0.040	1.525	0.006
Train	1.553	1.450	0.007		1.497	1.968	0.007
Swan	0.517	1.705	0.001		0.499	1.736	0.002

Table 3. Table 3: Time consumed by the proposed algorithm (Matlab edition) for atmosphere light estimation

Image	Size	Time consumed (s)
Forest	768*1024	0.108
Cones	384*465	0.172
Pumpkins	400*600	0.030
Hongkong	457*800	0.066
Train	400*600	0.029
Swan	557*835	0.056

Equations

$$I(x)=J(x)t(x)+A(1-t(x))$$

$$J^{dark}\left(x\right)=\underset{c\in \left\{r,g,b\right\}}{\min }\left(\underset{y\in \Omega \left(x\right)}{\min }\left(J^c\left(y\right)\right)\right)$$

$$J^{dark}\left(x\right)=\underset{c\in \left\{r,g,b\right\}}{\min }\left(\underset{y\in \Omega \left(x\right)}{\min }\left(J^c\left(y\right)\right)\right)\text{≈0}$$

$$\underset{y\in \Omega \left(x\right)}{\min }\left(\frac{I^c\left(y\right)}{A^c}\right)=\tilde{t}\left(x\right)\underset{y\in \Omega \left(x\right)}{\min }\left(\frac{J^c\left(y\right)}{A^c}\right)+\left(1-\tilde{t}\left(x\right)\right)$$

$$\underset{c}{\min }\left(\underset{y\in \Omega \left(x\right)}{\min }\left(\frac{I^c\left(y\right)}{A^c}\right)\right)=\tilde{t}\left(x\right)\underset{c}{\min }\left(\underset{y\in \Omega \left(x\right)}{\min }\left(\frac{J^c\left(y\right)}{A^c}\right)\right)+\left(1-\tilde{t}\left(x\right)\right)$$

$$\tilde{t}\left(x\right)=1-\underset{c}{\min }\left(\underset{y\in \Omega \left(x\right)}{\min }\left(\frac{I^c\left(y\right)}{A^c}\right)\right)$$

$$\tilde{t}\left(x\right)=1-\underset{c}{\mathrm{\omega min}}\left(\underset{y\in \Omega \left(x\right)}{\min }\left(\frac{I^c\left(y\right)}{A^c}\right)\right)$$

$$\left(L+\lambda U\right)t=\lambda \tilde{t}$$

$$J\left(x\right)=\frac{I\left(x\right)-A}{\max \left(t\left(x\right),t_0\right)}+A$$

$$X_C=\left\{x_{cand}|J^{dark}\left(x_{cand}\right)\in High\_Value\_Range\right\}$$

$$L=\{L_n|L_i\bigcap _{i\ne j}{L}_j=\varnothing ;\bigcup _{n=1}^5{L}_n=X_C;i,j,n=1,2,\cdots ,5\}$$

$$J=\sum _{n=1}^{K=5}\sum _{x_i\in L_n}\Vert x_i-{\mu }_n{\Vert }^2$$

$$L_{\infty }^R$$

$$L_{\infty }^C$$

$$L_{\infty }=mean\left(J\left(x_i^{{}^{\prime }}\right)\right)$$

Full text

{IEEEkeywords}

Statistics clustering, Atmosphere light, Transmissivity, Defogging, Image quality.

1 Introduction

Image defogging has become one hot spot in the domain of outdoor surveillance in the sense that smog represents more and more frequently a source of trouble to outdoor imaging and brings about, among other problems, low brightness and poor contrast, which greatly affect subsequent tasks of image segmentation, tracking, target detection, etc. Extensive research into this subject has been conducted by foreign and Chinese scholars,with a range of achievements, both theoretical and practical, resulted.[1, 2, 3, 4, 5, 6, 7]

Image defogging methods fall into two broad categories[8]: image enhancement-based enhancement, and physical model-based restoration. Image enhancement method, which does not account for the causes of image degradation, is effective in improving contrast of images taken on foggy days and bringing out their details, but entails some loss of image information. Image restoration method, on the other hand, tackles the problem by building a physical degradation model having regard to the image degradation process on foggy days. This method, attempting to invert the degradation process to arrive at fog-free images, produces clearer defogging effect and involves less information loss than image enhancement does, and so it is becoming a hot spot in the community of image defogging research.

As for restoration method, a physical model in general use is the atmosphere scattering model developed by Narasimhan et al.[9, 10, 11, 12, 13]. With this model, it is possible to get satisfactory restoration results by merely determining the transmissivity and the atmosphere light value of each point in the image. It is, however, not an easy job trying to determine these values. Okaley et al.[14] rely on radar installation to acquire the scene depth in order to establish the transmissivity function; Narasimhan et al.[9] extract the scene depth by use of two images taken under different weather conditions; Nayar et al.[15] seek to determine the scene depth and atmosphere light value by manually specifying the sky zone in the image. All these methods are too often not helpful in practical use because of certain restrictions. Over recent years, He came up with an algorithm[16]: dark original color a prior defogging, which attracts broad attention for its simplicity and efficacy. Dark original color a priori knowledge enables quick acquiring from the original image the transmissivity corresponding to each point, thus making real-time defogging possible, a feature at premium for outdoor surveillance. Its practical application, however, usually produces results that are troubled by color oversaturation, leading to image distortion. One significant reason is that the atmosphere light value is estimated too roughly. He’s estimation of atmosphere light is largely based on the dark channel image acquired from dark original color a priori. It works by selecting in the dark channel image some brighter points as candidate atmosphere light locations and then taking as atmosphere light points the brightest points of the pixels, in the original image, corresponding to these locations. This technique works well when the image contains a substantial area of sky, but may give rise to gross discrepancies in locating atmosphere light when the image contains no such sky area or when the image includes other large, bright, and white objects, as illustrated in Fig.1a, 1c, and 1e.

To cope with this problem, this paper presents an atmosphere light estimation method that is based on light source point cluster selection. The idea behind this method is performing clustering analysis of potential atmosphere light points to find out all possible atmosphere light zones, and then locating the atmosphere light by use of statistical information of potential points in each zone.

The rest of this paper is organized as follows. Section 2 gives an overview of atmosphere scattering model and dark original color a priori defogging algorithm and analyzes the drawbacks, and also their reasons, of the algorithm. In consideration of what has been described, Section 3 proposes an atmosphere light estimation method that is based on light source point cluster selection. Section 4 looks into a comparative experiment and substantiates the efficacy of the proposed algorithm from several indicators including atmosphere light location and image quality. Also in this section, the processing times of the algorithms are reported, confirming the feasibility of the proposed algorithm in actual applications.

2 Atmosphere Scattering Model

Narasimhan et al.[9, 10, 11, 12, 13] develop an atmosphere scattering model as below that describes the degradation process of images:

[TABLE]

where, $I$ is the intensity of the surveillance image, $J$ the light intensity of the scene, $A$ the atmosphere light at infinity, and $t$ the transmissivity. The first term, $J(x)t(x)$ , is called decay term, and the second term, $A\left(1-t\left(x\right)\right)$, is called atmosphere light term. The aim of defogging is to restore $J$ from out of $I$. To solve Eq.(1), we must determine $t(x)$ and $A$ in the first place.

2.1 Dark original color a priori knowledge

Dark original color a priori comes from statistics derived from outdoor images taken on fog-free days, and it gives us a piece of a priori knowledge: in most outdoor fog-free images there are always some pixels having a minimal light value in a certain color channel. This minimal light value is termed dark original color, and the corresponding pixels are known as dark original color pixels. The dark original color a priori knowledge may be defined by:

Let:

[TABLE]

where, $c$ represents a certain color channel, $J^{c}$ the component of image $J$ in this channel, and $\Omega(x)$ a square zone with pixel $x$ as its center.

If $J$ is an outdoor fog-free image, then for a certain pixel $x$ there exist always some pixels in its neighborhood $\Omega(x)$ such that $J^{dark}(x)\approx 0$, or:

[TABLE]

We now call $J^{dark}(x)$ dark original color, and refer to the above rule as dark original color a priori knowledge. This priori knowledge suggests that with fog-free images the $J^{dark}$ value is invariably minimal or close to 0.

2.2 Image defogging method based on dark original color a priori

2.2.1 Estimation method of transmissivity

In images tainted by fog, the intensity of $J^{dark}$ is higher than usual because of the superposition of white light component in the atmosphere, and for this reason such dark original colors are useful in estimating roughly the transmissivity $t(x)$ in foggy images.

Suppose the atmosphere light $A$ is given, and the transmissivity in a certain local neighborhood $\Omega(x)$ is constant. Apply the minimum operator to Eq.(1), then divide the result by $A$ to get:

[TABLE]

where, $c$ represents the color channel, $\tilde{t}(x)$ tells us that this is a rough estimate only.

Applying the minimum operator to color channel $c$ gives:

[TABLE]

But dark original color a priori knowledge says that in the case of fog-free outdoor images $J^{dark}$ should be close to 0. Substituting Eq.(2) into Eq.(3) gives a rough estimate of the transmissivity:

[TABLE]

It is interesting to note that images would look unnatural and the sense of depth is lost if the fog is removed completely. It is therefore necessary to introduce into Eq.(4) a constant $\omega\left(0<\omega\leq 1\right)$ ($\omega$ usually takes a value of 0.95) in order to keep some amount of foggy effect:

[TABLE]

The transmissivity given by the above equation is rather rough, and some block effect would be brought into the results. The algorithm in question uses an image matting algorithm[17] to get refined transmissivity $t(x)$. It is attained by solving the following linear equation:

[TABLE]

where, $\lambda$ is the correction factor, $L$ the Laplace matrix suggested by the matting algorithm, typically a large sparse matrix.

2.2.2 Estimation method of atmosphere light

The image defogging method based on dark original color a priori estimates atmosphere light $A$ by: organizing $J^{dark}$ in a descending order of brightness value, taking the top $0.1\%$ pixels as candidate atmosphere light points, comparing the brightness values of their corresponding points in the original image $I$, and taking the pixel with the highest brightness as $A$. This approach allows quick while accurate estimation of the atmosphere light location.

2.2.3 Image restoration

With refined transmissivity $t(x)$ and atmosphere light $A$ known, the restored image $J(x)$ may be found by the following equation:

[TABLE]

where, $t_{0}$ is a threshold designed to avoid too small a denominator, and it is typically set equal to 0.1.

2.2.4 Problems

Practical application reveals that this method sometimes sets atmosphere light $A$ on the brightest pixels, a place far off from real atmosphere light, rather than in a zone where the fog is the heaviest, for instance on white buildings or other large patches of white object. This observation is illustrated in Fig.1a, 1c, and 1e.

The method in discussion ranks $J^{dark}$ by brightness value and chooses a brighter but not the brightest pixel as candidate atmosphere light. This indeed precludes the brightest point from being chosen as atmosphere light point, but has its own problem nevertheless: $J^{dark}$ value is $\Omega\left(x\right)$-dependent, in other words it is influenced by $\Omega\left(x\right)$ size. This method would put the estimated atmosphere light on a bright object if this object happens to be larger than $\Omega\left(x\right)$in size. In Fig.1a, we can see that this method mistakenly puts the atmosphere light on the white building and, in Fig.1c and 1e, the atmosphere light is mistakenly put on the headlight and the white swan.

To cope with this problem, this paper introduces an atmosphere light estimation method that is based on light source point cluster selection.

3 Atmosphere Light Estimation Method Based on Light Source Point Cluster

Selection

With an eye to solving the problems associated with the above-described method, this paper comes up with an estimation method based on light source point cluster selection, which works by acquiring, by use of the dark original color a priori knowledge, the dark original color $J^{dark}\left(x\right)$ corresponding to image $J(x)$, ranking pixels $x$ by their brightness value, selecting the top $0.1\%$ pixels to form a set $X_{C}$ of candidate atmosphere light points:

[TABLE]

and then performing statistics clustering on point set $X_{C}$. This clustering is performed using the popular Euclidean distance $K-Means$ algorithm, with the clustering constant $K$ taking a value of 5:

[TABLE]

Thus the task is to find a point set such that the following equation is minimized:

[TABLE]

where, $x_{i}$ is a point from point cluster $L_{n}$, and $\mu_{n}$ the geometric center of point cluster $L_{n}$ .

When the clustering result, $L=\{L_{n}|n=1,2,\cdots,5\}$, is found, the point clusters $L_{n}$ are then ranked by the number $N_{n}$ of candidate light source points they contain. The point cluster with the highest point number, designated by $L_{n^{\prime}}$, is retained and the zone from which it comes is taken as the "sky zone", i.e., the zone of atmosphere light. Then, the geometric center, designated by $\mu_{n}$ , of all the candidate light source points $\{x_{i}^{{}^{\prime}}|x_{i}^{{}^{\prime}}\in L_{n^{\prime}}\}$ in $L_{n^{\prime}}$ is accepted as the location of atmosphere light, $(L_{\infty}^{R},L_{\infty}^{C})$, in the image, or:

[TABLE]

The mean of the brightness vectors of all the candidate light source points $\{x_{i}^{{}^{\prime}}|x_{i}^{{}^{\prime}}\in L_{n^{\prime}}\}$ in $L_{n^{\prime}}$ is taken as the atmosphere light brightness vector $L_{\infty}$. Or:

[TABLE]

Now that the atmosphere light brightness vector $L_{\infty}$ has been estimated, Eq.(5) enables us to determine the transmissivity $t(x)$, and finally Eq.(7) serves to find the defogging effect.

4 Experiment and Analysis

In this study, the dark original color was found using a window size $\Omega$ of $15\times 15$, with Ω in Eq.(5) taking a value of 0.95. Image matting was performed using the same algorithm as that in Ref.[17]. The experiment hardware platform was: Intel(R) Core(TM) i3-3220 @ 3.3GHz for CPU; 8GB DDR3 @ 1600MHz for memory. The software platform was: MATLAB R2016a 64-bit. All the algorithms were implemented using MATLAB code.

The old algorithm[16] relies on image matting to refine the transmissivity, a technique that in essence amounts to solving a large system of sparse linear equations, rather complex in both space and time. But this step is designed to remove the block effect arising from local windowing in estimating the transmissivity. Therefore, this study turns to a technique commonly used in practical engineering: first downscale the image to a suitable size, then refine the transmissivity by image matting, and finally restore the reduced image to its previous size using the refined transmissivity by means of cubic interpolation. Fig.2 compares the transmissivities estimated using the two methods and we see that they do not differ much in removing transmissivity-related block effect.

4.1 Location of atmosphere light

For comparison, the location estimated by the old method (denoted by red diamond) and that estimated by the proposed method (denoted by blue diamond) are shown in Fig.1. As shown in Fig.1a, the old algorithm mistakes the building, which is bright and constitutes a large portion of this image, for the atmosphere light; in contrast, the proposed algorithm locates correctly the atmosphere light in the sky zone(Fig.1b). In Fig.1c, the old algorithm makes the mistake of locating atmosphere light on the headlight, the brightest spot in the image, but the proposed algorithm accurately places atmosphere light on the upper left sky zone, despite that it makes up only a small portion of the image. Once more, in Fig.1e the old algorithm commits the error of taking the white swan as atmosphere light, but the proposed algorithm correctly recognizes the tree branch zone, the portion with the heaviest fog, as atmosphere light.

4.2 Defogging effect

Fig.3 provides a comparison of the images restored from smoggy images by the two algorithms, which have different atmosphere light estimation rationales. As is evident from this figure, relative to the old algorithm the proposed one produces effects that look more natural.

4.3 Quantitative indicators

In order to provide an objective evaluation of the defogging effect, three indicators, i.e., contrast enhancement ratio, EME, and blind assessment, are used to quantify the defogging effect.

Contrast is an important indicator of image quality because images with good contrast typically look clearer and better and those with a low contrast appear fuzzy or blurred. Table 1 lists the contrast enhancement ratios, for both the old and the proposed algorithms. It is obvious from the table that the proposed algorithm performs better in terms of contrast enhancement.

EME, an image quality indicator developed by Agaian et al.[18], is more consistent with human visual system (HVS), and here a higher EME stands for better image quality. Table 1 gives the EME values, again for both the old and the proposed algorithms. From this table we see that both algorithms improve the EME substantially in relation to the original images, and the proposed algorithm behaves better than the old one in terms of defogging effect EME in all but the swan image.

Blind assessment[19] is a quantitative indicator developed by Hautiere et al. for evaluating the restoration quality of images taken on foggy days. Its logarithmic image processing (LIP) model assesses the defogging effect from three perspectives: the ratio between new visible edges ($e$) before and after restoration (same below), the ratio between mean gradient ($r$), and ratio between black pixel proportion ($\sigma$). The higher the value of $e$ and $r$ and the lower the value of $\sigma$, the better the defogging effect after restoration. Table 2 provides a comparison of different indicators associated with the two algorithms and it appears that the proposed algorithm improves on the old one in all these indicators.

4.4 Operational speed

The proposed algorithm is designed to improve on images acquired by outdoor surveillance instruments and thus its operational efficiency represents an essential indicator as to its practical value. Table 3 records the operation time of the proposed algorithm and it may be concluded that this algorithm consumes minimal time when running on mainstream outdoor surveillance hardware platforms. For instance, it took only 0.108s when processing a forest image of 720P size (1080*720). It is useful to note that the tabulated results were delivered by an algorithm implemented using Matlab; if implemented in C/C++ or other similar languages, the algorithm is expected to, according to experience, register a minimum of tenfold improvement in computational efficiency, implying that a minimal load is created to the legacy outdoor surveillance equipment. Moreover, parallel computation typical of clustering makes it possible, by implementing an algorithm of parallel version, to bring additional improvement to computational efficiency.

5 Conclusion

This paper gives a theoretical analysis of the methodology used by an existing defogging algorithm for acquiring atmosphere light and, assisted by experimental observation, some drawbacks have been identified in this methodology, and then the reasons of these drawbacks are investigated. A technique for determining atmosphere light is subsequently proposed that is based on light source point cluster selection.

Experimental results indicated that the proposed algorithm produces more accurate information with respect to atmosphere brightness vector and location (Fig.1), the images restored using this method look more natural subjectively (Fig.3), and the objective image quality indicators are better (Table 1 and 2). Besides, the proposed algorithm consumes less time (Table 3), virtually giving no extra load to legacy outdoor surveillance systems.

The future effort of this study is on how to improve the algorithm such that its computational load is reduced and that it may be used with on-board surveillance systems and on other practical occasions.