Nighttime sky/cloud image segmentation

TL;DR

This paper introduces a superpixel-based segmentation method for nighttime sky/cloud images, addressing the challenges of low light and noise, and provides the first dedicated dataset for this task.

Contribution

The paper presents a novel superpixel-based segmentation algorithm specifically designed for nighttime sky/cloud images and releases the first dataset for this purpose.

Findings

The proposed method effectively segments nighttime sky/cloud images.

The new dataset facilitates further research in nighttime atmospheric imaging.

Experimental results demonstrate the algorithm's robustness under low-light conditions.

Abstract

Imaging the atmosphere using ground-based sky cameras is a popular approach to study various atmospheric phenomena. However, it usually focuses on the daytime. Nighttime sky/cloud images are darker and noisier, and thus harder to analyze. An accurate segmentation of sky/cloud images is already challenging because of the clouds' non-rigid structure and size, and the lower and less stable illumination of the night sky increases the difficulty. Nonetheless, nighttime cloud imaging is essential in certain applications, such as continuous weather analysis and satellite communication. In this paper, we propose a superpixel-based method to segment nighttime sky/cloud images. We also release the first nighttime sky/cloud image segmentation database to the research community. The experimental results show the efficacy of our proposed algorithm for nighttime images.

Tables

Table 1. Table 1 : Color spaces and components used in our analysis of nighttime images.

$c_1$	R	$c_4$	H	$c_7$	Y	$c_{10}$	$L^{\ast }$	$c_{13}$	$R/B$	$c_{16}$	$C$
$c_2$	G	$c_5$	S	$c_8$	I	$c_{11}$	$a^{\ast }$	$c_{14}$	$R-B$
$c_3$	B	$c_6$	V	$c_9$	Q	$c_{12}$	$b^{\ast }$	$c_{15}$	$\frac{B-R}{B+R}$

Table 2. Table 2 : Performance evaluation of different benchmarking algorithms. The average scores across all the images of the database are reported. The best performance according to each criterion is indicated in bold.

Methods	Precision	Recall	F-score	Error
Yang et al. [6]	0.98	0.69	0.79	0.14
Yang et al. [7]	0.90	0.19	0.26	0.40
Liu et al. [8]	0.68	0.93	0.77	0.23
Gacal et al. [9]	0.47	0.99	0.62	0.53
Proposed approach	0.95	0.76	0.83	0.13

Equations

$${\mathbf{X}}_f=[\mathbf{r}\mathbf{x}\mathbf{y}],$$

Full text

In this paper, we propose a superpixel-based method to segment nighttime sky/cloud images. We also release the first nighttime sky/cloud image segmentation database to the research community. The experimental results show the efficacy of our proposed algorithm for nighttime images.

**Index Terms— ** Superpixels, nighttime cloud segmentation, sky imagers, WAHRSIS, SWINSEG.

1 Introduction

Over the last few decades, there has been growing interest in geoscience to study clouds and analyze their various features. Satellite images are the conventional source for analyzing the formation of clouds, their movement, and other atmospheric properties. However, these images suffer from poor spatial and/or temporal resolution. Therefore, remote sensing analysts increasingly turn to ground-based sky imagers [1] that are able to produce images with significantly higher spatial resolution at much more frequent intervals. An accurate segmentation of the cloud mass from these images is of prime importance for any kind of cloud analysis.

Almost all existing works in sky/cloud image segmentation use color as the discriminatory feature to detect clouds in ground-based sky camera images. This is because the sky is predominantly blue during daytime owing to Rayleigh scattering of light. However, this assumption no longer holds during nighttime.

In the literature, most existing works consider only daytime images. Several color models are used in cloud segmentation [2, 3, 4, 5]. Yang et al. (2009) used the Otsu threshold on red-blue difference images [6]. Yang et al. (2010) employed a local method [7] for cloud segmentation, which uses an adaptive threshold on sub-images of the sky. Recently, Liu et al. [8] used a superpixel approach and generated corresponding thresholds for each superpixel to compute the final binary image. The local thresholds of these superpixels are empirically evaluated based on a set of case-based decisions, thereby generating a matrix of thresholds.

Cloud segmentation in the nighttime sky is often neglected. Yet, analyzing clouds at nighttime helps in several areas viz. weather reporting and prediction, aviation, and satellite communication. Only very recently, Gacal et al. in  [9] explored the nighttime cloud detection using ground-based sky cameras. Their method uses a constant fixed threshold in the gray-scale channel of the cloud images, captured over Manila region in the Philippines.

Addressing this gap in the literature, we propose an efficient nighttime cloud segmentation approach. In this paper, we perform a careful selection of color spaces and components, and thereby propose a superpixel-based approach to efficiently segment nighttime sky/cloud image. We also present a new nighttime sky/cloud image segmentation database, which we use for benchmarking our approach.

2 Nighttime Cloud Segmentation

In this section, we present our detailed segmentation framework for nighttime cloud segmentation.111 The source code of all simulations in this paper is available online at https://github.com/Soumyabrata/nighttime-imaging. Our approach is illustrated in Fig. 1. Using an input image as shown in Fig. 1(a), we obtain a suitable color channel (in this example the red-blue difference), shown in Fig. 1(b). We then perform an over-segmentation of this image using a superpixel technique; the over-segmented image is shown in Fig. 1(c). Thereafter, we convert it into an indexed image as shown in Fig. 1(d) using a quantization procedure. Subsequently, we cluster these superpixels into two clusters – cloud and sky. The resulting binary segmented image is shown in Fig. 1(e).

Unlike the conventional daytime sky/cloud image, nighttime images include several additional imaging challenges. Figure 2 shows a sample day- and night-time image captured by our high resolution sky camera. Unlike the clear blue sky and white clouds seen during daytime, the night sky and clouds appear more reddish. This is mainly because of increasing light pollution in urban areas like Singapore. In a cloudy night-sky, the longer wavelengths of the artificial light are scattered back to the earth’s surface, providing a reddish tinge to the night sky. More details on this phenomenon can be found in a recent study [10]. Moreover, nighttime images are more noisy and blurry in nature, with less distinctive features, owing to the higher exposure time and lower f-numbers of the camera lens.

2.1 Notation

Let us suppose that an image is represented by $\mathbf{X}\in{\rm I\!R}^{m\times n\times 3}$. We perform an analysis of different color spaces and components that are frequently used in cloud segmentation. Table 1 shows the $16$ color channels, represented by $c_{1-16}$, that are used in our analysis.

It consists of channels from color models such as $RGB$, $HSV$, $YIQ$, $L^{*}a^{*}b^{*}$, various combinations of red and blue color channels, and chroma ($C$). We choose the most discriminatory color channel amongst these channels for our segmentation approach (see Section 4.1). Let us denote this discriminatory channel by $\mathbf{c_{*}}\in{\rm I\!R}^{m\times n}$. We represent the feature vector used during superpixel clustering as $\mathbf{X}_{f}$, and the value of the channel $\mathbf{c_{*}}$ at the image co-ordinates $(x,y)$ is $r$. Finally, the number of superpixels is $P$.

2.2 Proposed Algorithm

As described in Fig. 2, the nighttime images are noisy in nature, and pixel-wise classification into day- and night- pixel is not conducive. Therefore, we employ a superpixel-based method that groups similar pixels together in a single superpixel, while respecting cloud boundaries. We use the superpixel generation algorithm, Simple Linear Iterative Clustering (SLIC) [11] in our proposed nighttime image segmentation approach. Instead of the default CIE LAB color space used by SLIC, we use the discriminatory channel $\mathbf{c_{*}}$ for the superpixel clustering. The clustering is performed on the $3D$ feature space $\mathbf{X}_{f}$. We calculate this feature as:

[TABLE]

where the vectors $\mathbf{r}$, $\mathbf{x}$, $\mathbf{y}$ represent the value, x- and y-coordinate for the vectorized form of image $\mathbf{c_{*}}$. This $3D$ feature is used for superpixel clustering, similar to the $5D$ LAB clustering technique used in SLIC approach [11]. We thereby obtain an over-segmented image on the channel $\mathbf{c_{*}}$. An example is illustrated in Fig. 1(c).

The generated superpixels in Fig. 1(c) respect the sky/cloud image boundaries very well, under the assumption that each superpixel contains only sky or only cloud pixels. Therefore, we compute the indexed image by mapping each pixels inside a superpixel to a common value $\bar{r}$. This common quantity $\bar{r}$ for a superpixel is estimated by averaging all the $r$ values in that corresponding superpixel. The indexed image is shown in Fig. 1(d). This vector quantization in the color channel $\mathbf{c_{*}}$ is performed to reduce the color depth of the discriminatory channel, and is inspired by Puzicha et al. [12].

Finally, we perform an unsupervised k-means clustering on this quantized channel, whose feature vector comprises the average value $\bar{r}$ for the individual superpixels. We perform this clustering into sky and cloud clusters. The cluster center with lower value is assigned sky label, and the one with higher value is assigned cloud label. The final binary mask obtained via k-means clustering is illustrated in Fig. 1(e).

In our proposed approach, the only input parameter is the number of superpixels $P$ used in the channel $\mathbf{c_{*}}$, where $P\in\mathbb{N},1\leq T\leq mn$. Therefore, the approximate size of a superpixel is $(mn/P)$ pixels. The constant $P$ is an indication of the degree of over-segmentation of the image. A large $P$ facilitates capturing fine features of the input image. We should set $P$ to a large enough value, such that each superpixel contain only sky or cloud pixels. More discussion on the value of $P$ can be found in Section 4.

3 Database

While there has been extensive work on daytime sky/cloud image segmentation, nighttime sky imaging is rare. Therefore, there are no publicly available nighttime sky/cloud image segmentation databases. We therefore created a new database specifically for this purpose. In analogy to our popular SWIMSEG database with daytime sky images published earlier [13], we refer to this database as Singapore Whole sky Nighttime Imaging SEGmentation Database (SWINSEG).222 The SWINSEG dataset is available for download at http://vintage.winklerbros.net/swinseg.html.

SWINSEG consists of $115$ diverse images with a few sample images shown in Fig. 3. The images were captured using a ground-based whole sky imager called Wide Angle High Resolution Sky Imaging System (WAHRSIS) [14], designed and deployed at Nanyang Technological University in Singapore. The images of SWINSEG were captured during the period January to December 2016. The images were hand-picked such that most of the variation in the image types are captured in the dataset. Several factors such as time of the image capture, cloud coverage, and seasonal variations were considered in the selection process. The distribution of images in SWINSEG according to cloud coverage percentages is shown in Fig. 4.

The extreme wide-angle fish-eye lens of WAHRSIS allows us to capture almost the entire hemisphere. The distortions introduced by this lens need to be undone before further processing. We generate undistorted pictures of a smaller subsection of the captured images using a ray tracing approach [13] by considering an imaginary camera (with a standard lens) at the center of the hemisphere, which points towards a user-defined direction. In order to simulate this camera, we consider a target image of dimension $500\times 500$, with altitude of the virtual image plane at $150$m, where each pixel represents an incident light ray. All those rays will intersect the hemisphere at a certain point and then converge towards the center. The value of a pixel is then equal to the color of the hemisphere at its intersection point. Figure 5 shows the output image as well as the lines corresponding to its borders and diagonals on the original image.

The segmentation ground truth for SWINSEG was generated in consultation with experts from the Singapore Meteorological Service.

4 Experimental Evaluation

In order to present an objective evaluation of the proposed algorithm, we report segmentation accuracy in terms of precision, recall, F-score, and error rate. Suppose TP, TN, FP and FN denote the true positive, true negative, false positive and false negative samples of a segmented image, then $\mbox{Precision}=TP/(TP+FP)$ and $\mbox{Recall}=TP/(TP+FN)$. The F-score is the harmonic mean of Precision and Recall. The error rate is the ratio of pixels classified incorrectly.

We also use Receiver Operating Characteristics (ROC) curves to select the optimal color channel for nighttime segmentation. The ROC curve is a graphical plot between true positive rate and false positive rate, as the threshold is gradually changed. We compute the area between the ROC curve and the random classifier slope. These scores are averaged over all images in the dataset.

4.1 Color Channel Selection

For an efficient nighttime image segmentation, it is important to choose the most discriminatory color channel $\mathbf{c_{*}}$ in our approach. We select the best color channel using the area between the ROC curve and the random classifier slope, to identify the discriminatory capability of the respective color channels. This is shown in Fig. 6.

The best color channel for nighttime sky/cloud image segmentation is $c_{14}$ ($R-B$), closely followed by $c_{12}$ ($b^{*}$) and $c_{8}$ ($I$). The ranking of color channels is somewhat different from the one for daytime sky/cloud image segmentation [13]; in particular, those channels involving red are better for night sky/cloud segmentation.

As discussed in Section 2, the over-segmentation is dependent on the number of superpixels $P$ in an image. $P$ should be large enough to capture all the essential details of the sky/cloud image. Ideally each superpixel should contain either only sky or only cloud. We experimentally set $P=100$. This makes the average size of a superpixel for our test images in the SWIMSEG dataset about $50^{2}$ pixels, which appears to be a good size for this purpose.

4.2 Benchmarking

In the literature, thresholding techniques in various color spaces are widely used in cloud segmentation. As discussed in Section 1, we benchmark our proposed framework with several state-of-the-art methods, including Yang et al. [6], Yang et al.  [7], Liu et al. [8] and Gacal et al. [9].

Table 2 provides an objective evaluation of these algorithms. We observe that methods such as Gacal et al. and Liu et al. achieve very high recall but low precision (i.e. they are too optimistic about what is a cloud). The method from Yang et al. [6] on the other hand has a high precision and low recall (i.e. too optimistic about sky regions). A good segmentation method must have both high precision and high recall, achieving a balance between sky and cloud detection; this is indicated by the F-score and error rate. Our proposed method outperforms other approaches, based on error rates, and has the highest F-score value.

5 Conclusion

We have presented a robust approach for the segmentation of nighttime sky/cloud images obtained from ground-based sky cameras. Our proposed method is based on superpixel segmentation of the image and is entirely threshold-free. It outperforms other state-of-the-art algorithms. We also release the first nighttime sky/cloud image segmentation database to the research community. Our future work includes increasing the SWINSEG dataset size.