Segmentation-Aware Hyperspectral Image Classification

TL;DR

This paper introduces a unified hyperspectral image classification approach combining deep neural networks with segmentation-aware superpixels, achieving state-of-the-art results and improving performance with limited training data.

Contribution

The paper presents a novel integration of segmentation-aware superpixels with deep residual networks for hyperspectral image classification.

Findings

Achieves state-of-the-art results on benchmark datasets.

Segmentation-aware superpixels significantly improve classification with limited training data.

The method effectively combines spectral and spatial information.

Abstract

In this paper, we propose an unified hyperspectral image classification method which takes three-dimensional hyperspectral data cube as an input and produces a classification map. In the proposed method, a deep neural network which uses spectral and spatial information together with residual connections, and pixel affinity network based segmentation-aware superpixels are used together. In the architecture, segmentation-aware superpixels run on the initial classification map of deep residual network, and apply majority voting on obtained results. Experimental results show that our propoped method yields state-of-the-art results in two benchmark datasets. Moreover, we also show that the segmentation-aware superpixels have great contribution to the success of hyperspectral image classification methods in cases where training data is insufficient.

Tables

Table 1. Table 1 : Overal accuracies of University of Pavia and Indian Pines datasets.

	Raw	SVM[11]	SAE[12]	EMP[13]	LeNet[14]	CNN[15]	SSUN[6]	SSRN[5]	Our Method
Uni. of Pavia	90.80	90.58	94.25	97.61	97.73	98.64	99.46	99.79	99.93
	$\begin{array}{c}+\\ -\end{array}$ 0.37	$\begin{array}{c}+\\ -\end{array}$0.47	$\begin{array}{c}+\\ -\end{array}$0.18	$\begin{array}{c}+\\ -\end{array}$0.19	$\begin{array}{c}+\\ -\end{array}$0.96	$\begin{array}{c}+\\ -\end{array}$0.20	$\begin{array}{c}+\\ -\end{array}$0.32	$\begin{array}{c}+\\ -\end{array}$0.09	$\begin{array}{c}+\\ -\end{array}$0.02
Indian Pines	79.68	81.67	85.47	92.22	96.01	97.41	98.40	99.19	99.38
	$\begin{array}{c}+\\ -\end{array}$ 0.87	$\begin{array}{c}+\\ -\end{array}$0.65	$\begin{array}{c}+\\ -\end{array}$0.58	$\begin{array}{c}+\\ -\end{array}$0.71	$\begin{array}{c}+\\ -\end{array}$0.87	$\begin{array}{c}+\\ -\end{array}$0.43	$\begin{array}{c}+\\ -\end{array}$0.37	$\begin{array}{c}+\\ -\end{array}$0.26	$\begin{array}{c}+\\ -\end{array}$0.06

Table 2. Table 2 : Overal accuracies of University of Pavia dataset for different percentage of training data.

	0.5%	5%	10%
SSRN	90.01	99.24	99.79
	$\begin{array}{c}+\\ -\end{array}$0.02	$\begin{array}{c}+\\ -\end{array}$0.03	$\begin{array}{c}+\\ -\end{array}$0.09
Our Method	90.47	99.36	99.93
	$\begin{array}{c}+\\ -\end{array}$0.04	$\begin{array}{c}+\\ -\end{array}$0.06	$\begin{array}{c}+\\ -\end{array}$0.02

Table 3. Table 3 : Overal accuracies of Indian Pines dataset for different percentage of training data.

	0.5%	5%	20%
SSRN	30.52	91.99	99.19
	$\begin{array}{c}+\\ -\end{array}$0.13	$\begin{array}{c}+\\ -\end{array}$0.14	$\begin{array}{c}+\\ -\end{array}$0.26
Our Method	31.54	92.14	99.38
	$\begin{array}{c}+\\ -\end{array}$0.20	$\begin{array}{c}+\\ -\end{array}$0.12	$\begin{array}{c}+\\ -\end{array}$0.06

Equations

$$Z=f(X_H,W)$$

$$S=g(X_{RGB},n)$$

$$Y_{i,j}=\varphi (Z_{i,j},S_k)$$

$$\varphi (Z_{i,j},S_k)=\underset{t}{\mathrm{argmax}}\frac{{\sum }_{t=1}^C{\sum }_{p=1}^{m\times n}\left[1_p^{S_k}\times 1_{Z_p}^{C_t}\right]}{{\sum }_{p=1}^{m\times n}\mid 1_p^{S_k}\mid }$$

Full text

**Index Terms— ** hyperspectral image classification, deep neural networks, segmentation, superpixel, deep affinity networks

1 Introduction

Hyperspectral images are obtained by dividing the electromagnetic spectrum into several narrow bands that can be extended beyong the visible light. Thanks to this feature, strong spectral classification of surfaces and objects are possible through hyperspectral data. Hyperspectral imaging technology is currently used in many areas such as target detection [1], plant species detection [2, 3], and water property detection [4]. In these studies, various machine learning techniques have been used. More recently, convolutional neural network (ConvNet) based approaches have lead to great advances in some hyperspectral imaging problems, especially in target classification [5, 6].

Deep learning based approaches require large amount of labeled images (e.g. ImageNet [7]) in the classification problem. In the field of hyperspectral imaging, it is not possible to obtain large amount of images due to sensor and preprocessing costs. Moreover, deep neural network approaches for hyperspectral classification problem use certain amount of spectral signatures collected from same image with test set for training purpose and it leads to biased deep classification models. Therefore, it is necessary to focus on transfer learning approaches in hyperspectral imaging area.

It is difficult to transfer information in the spectral sense, but it is possible to transfer spatial information which are useful for encoding context information from three-band datasets [8]. In this paper, we try to transfer spatial information by using pixel affinity network based segmentation-aware superpixels [9]. Pixels that are spatially close each other and have similar texture or color information will belong to the same class and according to this hypothesis, we can assume that pixels that are located in a segmentation-aware superpixel belong to the same class. Therefore, we can correct the erroneous class predictions obtained from deep neural networks by conducting dominance analysis on the superpixel groups. In this model, pixel affinity network is trained on some images of BSDS500 dataset [10]. As a deep convolutional model, we use an architecture which is identical to that described in [5]. The proposed deep model learns spectral and spatial representations separately by using two sequential residual blocks. We evaluate the proposed method on University of Pavia and Indian Pines datasets and experimental results show that our propoped method yields state-of-the-art results in these two benchmark datasets.

To sum up, we proposed an unified classification method that consists of a deep neural network which uses spectral and spatial information together with residual connections and pixel affinity network based on segmentation-aware superpixels together. Our unified framework is summarized on Figure 1. Our main contributions related to the classification of hyperspectral images as follows:

We transfer the spatial information from three-band datasets (e.g. BSDS500) for encoding context information. 2. 2.

We learn with limited spectral training data by using this spatial information and compare our model with state-of-the-art models in the same setting. 3. 3.

The proposed method obtains the state-of-the-art performance among hyperspectral image classification algorithm for two benchmark datasets.

In the rest of the paper, we first describe the proposed method in Section 2 and then talk about experimental setup, and compare experimental results in Section 3. Finally, we will summarize the motivation behind this paper and discuss other things in Section 4.

2 Method

In this section we describe the proposed method. Our classification model needs three-dimensional hyperspectral data cube and RGB version of it to generate a classification map in two subsections: (1) RGB image is used as an input for segmentation-aware superpixels, and (2) hyperspectral data is used as an input in deep neural network. In order to model this process, we first train a deep neural network which uses spectral and spatial information together with residual connections.

The neural network predicts a classification result of hyperspectral data. This process is carried out in parallel with the generation of the segmentation-aware superpixels over the RGB image. All the spectral signatures within the superpixel will have the same color and texture information and therefore belong to the same class. We use them to compensate for the incorrect classification results obtained from the deep neural network. For this purpose, we perform a dominance analysis within each superpixel and transfer the dominant class information to all pixels within the superpixel. Computing superpixels with learned pixel affinities provides an opportunity to preserve object boundaries.

Our approach can be defined as follows: let $X_{H}$ be the hyperspectral data cube, $X_{RGB}$ be the RGB version of $X_{H}$, $Z$ be the classification map obtained from deep neural network and $Y$ be the final classification map. We define a function $f(x,W):X_{H}\rightarrow R^{m\times n\times\lambda}$ for classification results that are obtained from deep neural network:

[TABLE]

where $W$ is weights for trained deep network which uses spectral and spatial information together with residual connections. The neural network used in this paper is identical to the model described in [5].

We define an another function $g(x,n):X_{RGB}\rightarrow R^{m\times n}$ for generating segmentation-aware superpixels:

[TABLE]

where $n$ is the number of expected superpixels. Here, $g$ is a pixel affinity network based segmentation-aware superpixel algorithm. Pixel affinity network is trained on BSDS500 dataset [10] for preserving object boundaries by using segmentation-aware loss in superpixel generation. The details of the pixel affinity network and segmentation-aware superpixels are described in [9].

We compute the dominant class information for each superpixel and apply this class information all pixels within the related superpixel:

[TABLE]

where $Y_{i,j}$ is the final classification result of pixel location $(i,j)$, and $Z$ is the classification result of the deep neural network. Moreover, $S_{k}$ represent the superpixel group that consist of $i,j$ coordinate. Here, $\phi(Z_{i,j},S_{k})$ is defined as follows:

[TABLE]

where $C_{t}$ represents the candidate class, $p$ represents the current pixel and $\mathbb{1}_{p}^{S_{k}}$ notation represents the scalar value that 1 if the selected pixel is located in $S_{k}$ superpixel group, otherwise 0. Moreover, $\mathbb{1}_{Z_{p}}^{C_{t}}$ represents the value that 1 if the classification result of pixel $p$ equals to the label of the class $C_{t}$, otherwise 0.

3 Experiments

In this section, we present the effectiveness of the proposed approach by considering two different hyperspectral image classification datasets: University of Pavia and Indian Pines.

In the rest of the section, we first describe the datasets in Section 3.1. Then, we give the implementation details and experimental results in Section 3.2. Finally, we will discuss ablative studies on training data in Section 3.3.

3.1 Datasets

University of Pavia dataset has $610\times 340$ pixels with 9 different land-cover class types. It is collected from the reflective optics system imaging spectrometer (ROSIS-03) and its spatial resolution is 1.3m by pixels. In our experiments, we use $0.5\%$, $5\%$ and $10\%$ of total pixels for training purpose separately.

Indian Pines dataset has $145\times 145$ pixels with 16 different vegetation class types. It is acquired by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) and its spatial resolution is 20m by pixels. For the experiments of the Indian Pines dataset, we use $0.5\%$, $5\%$ and $20\%$ of number of total pixels for training purpose separately.

3.2 Experimental Results

In our experiments, we select number of superpixels as $n=10.000$ to obtain fine-grained superpixel groups. We first compare the performance of the proposed approach with other state-of-the-art results by running the proposed method with same percentage of training data. In these experiments, training of both University of Pavia and Indian Pines models are completed in $40$ epoches. Moreover, we select batch size as 16 and learning rate as 0.001.

We compare our proposed approach with SVM[11], SAE[12], EMP[13] , LeNet[14], CNN[15], SSUN[6] and SSRN[5] methods. The overall accuracy results are given in Table 1. In these experiments, we use $10\%$ of total pixels in University of Pavia and $20\%$ of total pixels in Indian Pines datasets for training purpose. We observe that our method obtain state-of-the-art results on both datasets.

The results on Table 1 show the importance of the context information in hyperspectral image classification problem. Moreover, these results also show that we can provide context information with segmentation-aware superpixels which are trained on three-band computer vision datasets. We share our final classification map for University of Pavia dataset on Figure 2.

3.3 Ablative Studies on Training Data

In this section, we will discuss ablative studies on training data. One of the contributions we have made in this paper is we show that spatial information which are obtained from segmentation-aware superpixels are very useful in situations where there is a limited training set for hyperspectral image classification.

We compared our method with the SSRN [5] by using various amounts of training set. We use University of Pavia and Indian Pines datasets during the experiments and we share our results on Table 2 and Table 3. From these results, we see that segmentation-aware superpixels contribute to the classification models that have a certain level of success.

4 Conclusion

In this paper, we proposed a hyperspectral image classsification method which use pixel affinity network based segmentation-aware superpixels and deep neural network in a unified framework. Several experimental results show that the proposed method yields state-of-the-art results in two benchmark datasets. Also, ablative studies show that the segmentation-aware superpixels have great contribution to the success of classification in cases where training data are insufficient.