Segmenting Hyperspectral Images Using Spectral-Spatial Convolutional Neural Networks With Training-Time Data Augmentation

TL;DR

This paper introduces a spectral-spatial CNN for hyperspectral image classification that leverages data augmentation to improve accuracy and real-time performance despite limited training data.

Contribution

The paper proposes a novel spectral-spatial CNN with training-time data augmentation, enhancing classification accuracy in hyperspectral imaging with limited ground-truth data.

Findings

Outperforms existing spectral-spatial methods

Achieves real-time hyperspectral classification

Effective with limited training data

Abstract

Hyperspectral imaging provides detailed information about the scanned objects, as it captures their spectral characteristics within a large number of wavelength bands. Classification of such data has become an active research topic due to its wide applicability in a variety of fields. Deep learning has established the state of the art in the area, and it constitutes the current research mainstream. In this letter, we introduce a new spectral-spatial convolutional neural network, benefitting from a battery of data augmentation techniques which help deal with a real-life problem of lacking ground-truth training data. Our rigorous experiments showed that the proposed method outperforms other spectral-spatial techniques from the literature, and delivers precise hyperspectral classification in real time.

Tables

Table 1. TABLE I: The results obtained using all variants of the investigated methods for Salinas Valley.

Algorithm	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15	C16	OA	AA
3D-CNN	96.49	75.15	39.89	61.61	51.78	79.21	76.81	74.84	78.14	85.69	71.56	76.49	80.86	62.15	61.80	33.00	69.72	69.09
3D-CNN-Rotate	90.83	85.00	39.64	74.26	54.51	79.44	78.42	78.00	76.91	84.55	67.96	79.83	97.56	75.40	62.09	34.62	72.01	72.44
3D-CNN-Flip	94.63	80.41	38.85	72.09	49.43	79.26	78.11	76.09	76.01	83.15	62.62	79.61	85.03	53.64	55.10	32.64	69.04	68.54
3D-CNN-Zoom	91.55	71.35	38.89	58.61	53.01	79.60	76.37	73.75	76.82	85.67	69.19	73.69	89.45	53.44	60.41	33.34	68.47	67.82
3D-CNN-Mixed	95.12	80.70	40.79	74.82	53.36	79.20	78.99	75.77	79.50	86.48	64.53	79.73	97.16	67.40	60.76	32.43	71.38	71.67
BASS	97.78	85.01	59.30	78.70	49.23	79.05	76.76	70.62	86.31	83.47	73.64	77.30	68.50	66.41	58.03	31.39	70.93	71.35
BASS-Rotate	97.65	90.12	53.04	79.27	52.98	79.93	79.23	75.99	83.32	88.21	72.85	82.05	85.94	75.51	58.61	34.27	73.38	74.31
BASS-Flip	98.49	83.28	53.02	78.96	48.12	79.78	78.48	75.06	81.84	85.08	74.38	80.56	74.70	76.43	55.41	30.90	71.29	72.16
BASS-Zoom	97.40	87.74	50.79	78.82	52.80	79.36	78.18	74.43	83.11	84.87	73.68	78.55	69.79	67.46	57.07	34.02	71.62	71.75
BASS-Mixed	95.42	88.22	50.15	78.87	50.13	79.68	79.01	75.16	85.28	86.54	74.15	81.92	83.26	78.87	61.77	32.21	73.28	73.79
Ours	93.22	90.48	47.57	78.97	57.35	79.48	79.13	75.77	79.71	88.59	74.68	83.48	94.53	85.36	62.52	32.47	73.72	75.21
Ours-Rotate	95.30	90.30	50.35	78.89	56.59	79.56	79.38	76.37	86.89	88.01	74.79	84.77	98.55	87.76	65.67	34.97	75.54	76.76
Ours-Flip	95.95	89.58	47.63	78.98	57.87	79.64	79.19	76.64	84.50	87.92	73.79	85.18	94.29	86.54	62.94	34.79	74.77	75.96
Ours-Zoom	95.65	81.47	47.61	78.54	55.22	79.65	78.66	75.73	86.61	88.24	76.37	85.14	98.94	85.41	62.68	33.36	74.04	75.58
Ours-Mixed	89.95	85.48	47.25	78.78	57.20	79.64	79.30	76.36	83.51	88.23	74.91	84.09	97.83	85.56	63.67	34.83	74.13	75.41
How to read this table: The globally best result (across all techniques) is boldfaced, and the background of the globally worst cell is red.
For each method (3D-CNN, BASS, and our deep network) separately, the background of the best variant is green, and the background of the worst variant is gray.

Table 2. TABLE II: The results obtained using all variants of the investigated methods for Indian Pines. The meanings of colors as in Table  I .

Algorithm	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15	C16	OA	AA
3D-CNN	5.00	33.70	28.30	17.88	51.32	60.18	0.00	65.99	1.67	53.06	54.27	23.20	65.87	77.01	37.95	37.94	48.89	38.33
3D-CNN-Rotate	6.25	49.53	46.13	44.73	65.92	90.42	0.00	84.73	26.53	68.79	60.12	27.87	84.07	79.79	36.45	43.85	60.34	50.95
3D-CNN-Flip	8.75	42.18	31.59	36.62	52.25	81.79	0.00	82.05	20.83	62.69	60.96	23.40	71.69	82.66	33.85	49.67	55.98	46.31
3D-CNN-Zoom	5.18	36.19	29.73	23.48	51.18	81.86	0.00	76.07	25.21	58.19	67.88	21.52	65.50	84.54	23.56	42.83	55.25	43.31
3D-CNN-Mixed	12.32	45.18	40.46	40.18	62.46	84.71	0.00	80.76	26.42	66.34	60.37	27.50	80.85	82.06	36.36	49.18	58.52	50.44
BASS	0.00	29.68	16.49	29.45	22.58	59.39	0.00	80.43	0.00	26.99	65.47	16.28	15.71	82.32	21.61	23.89	45.30	30.64
BASS-Rotate	1.00	40.66	16.38	32.87	30.95	70.55	0.00	82.57	0.00	41.57	51.77	20.63	27.02	83.85	21.36	34.14	46.82	34.69
BASS-Flip	1.00	36.08	19.38	26.70	24.13	67.54	0.00	78.72	0.00	36.92	55.56	17.21	27.20	85.31	27.03	32.73	46.38	33.45
BASS-Zoom	0.00	32.31	23.54	25.27	28.31	57.17	0.00	82.01	0.00	41.74	50.21	18.26	20.65	81.97	22.43	27.82	44.35	31.98
BASS-Mixed	1.61	29.37	25.23	33.46	27.81	61.32	0.00	78.71	0.00	47.85	53.19	18.34	37.40	83.24	28.71	40.80	46.33	35.44
Ours	5.18	43.14	27.72	24.74	56.02	83.89	0.00	72.59	5.85	53.12	65.39	20.35	55.49	81.00	33.74	35.22	54.93	41.47
Ours-Rotate	5.18	52.80	38.00	40.58	71.54	93.65	0.00	69.96	10.56	63.07	65.76	29.44	89.07	83.00	40.92	40.00	61.33	49.60
Ours-Flip	4.46	43.54	32.76	37.43	60.21	87.57	0.00	66.43	3.33	59.62	62.64	22.13	56.23	82.08	36.54	34.73	56.06	43.11
Ours-Zoom	1.00	40.42	29.22	21.57	58.49	84.53	0.00	70.48	1.00	59.43	67.34	20.00	60.32	80.55	29.36	33.88	55.46	41.07
Ours-Mixed	1.25	51.55	35.65	42.43	67.20	90.43	0.00	74.90	5.56	61.10	64.60	24.38	77.20	86.00	34.98	38.23	60.01	47.22

Table 3. TABLE III: The results obtained using all variants of the investigated methods for Pavia University. The meanings of colors as in Table  I .

Algorithm	C1	C2	C3	C4	C5	C6	C7	C8	C9	OA	AA
3D-CNN	90.66	81.85	41.92	93.02	59.79	25.20	0.00	70.18	79.03	70.07	60.18
3D-CNN-Rotate	92.96	82.39	41.96	93.46	59.93	26.38	0.00	73.98	79.89	71.19	62.93
3D-CNN-Flip	91.15	82.45	39.63	92.57	59.97	25.44	0.00	75.37	79.40	70.77	60.66
3D-CNN-Zoom	89.12	83.18	41.09	92.34	59.91	25.01	0.00	70.82	79.32	70.37	60.09
3D-CNN-Mixed	92.45	81.49	44.07	94.13	59.99	26.14	0.00	74.52	79.39	70.88	61.35
BASS	91.27	79.34	33.96	90.47	59.86	24.45	0.00	72.94	77.96	68.62	58.92
BASS-Rotate	93.55	78.49	30.43	92.29	59.96	25.21	0.00	81.51	79.26	69.44	60.01
BASS-Flip	91.13	82.13	41.07	88.84	49.83	17.00	0.00	74.87	72.49	68.82	57.48
BASS-Zoom	92.90	77.96	36.55	91.53	59.99	25.16	0.00	71.49	77.31	68.41	59.21
BASS-Mixed	92.52	78.25	34.61	92.47	59.82	25.35	0.00	82.54	78.18	69.48	60.42
Ours	94.13	77.17	43.38	92.37	59.90	23.52	0.00	75.79	77.47	68.83	60.41
Ours-Rotate	95.76	76.78	45.86	93.58	59.89	22.83	0.00	76.23	78.22	69.11	61.02
Ours-Flip	93.96	75.61	45.93	92.47	59.93	22.90	0.00	79.93	77.03	68.56	60.86
Ours-Zoom	94.04	76.98	46.20	91.88	59.96	22.60	0.00	76.11	76.74	68.76	60.50
Ours-Mixed	94.79	77.05	45.43	92.87	59.97	23.35	0.00	80.57	77.88	69.44	61.32

Table 4. TABLE IV: The ranking of all methods (averaged across all sets; AR), alongside the training times (s), and the inference time of classifying one test sample (ms). The best ranking is boldfaced.

		Training time			Inference time
Algorithm$\downarrow$	AR	Sa	IP	PU	Sa	IP	PU
3D-CNN	9.33	77.38	31.38	19.23	0.17	0.08	0.08
3D-CNN-Rotate	3.67	112.99	46.94	32.34	0.17	0.08	0.08
3D-CNN-Flip	7.67	125.84	47.96	30.72	0.17	0.09	0.08
3D-CNN-Zoom	9.00	102.23	42.54	34.36	0.17	0.08	0.08
3D-CNN-Mixed	5.33	125.81	44.35	30.39	0.17	0.08	0.08
\hdashlineBASS	13.33	339.60	207.64	75.01	0.29	0.27	0.12
BASS-Rotate	8.33	585.20	241.12	85.78	0.29	0.27	0.12
BASS-Flip	12.33	586.21	346.53	119.61	0.29	0.27	0.12
BASS-Zoom	13.00	619.48	339.39	81.54	0.29	0.27	0.12
BASS-Mixed	8.50	552.16	316.34	100.68	0.29	0.27	0.12
\hdashlineOurs	8.00	157.32	103.69	29.58	0.18	0.16	0.09
Ours-Rotate	3.67	248.24	156.00	48.26	0.18	0.16	0.09
Ours-Flip	6.17	255.40	161.30	40.66	0.18	0.16	0.09
Ours-Zoom	7.50	218.86	155.68	33.87	0.18	0.16	0.09
Ours-Mixed	4.17	232.53	163.47	36.12	0.18	0.16	0.09

Table 5. TABLE V: Wilcoxon tests show that the differences between our deep network and 3D-CNN are statistically important for almost all variants (the p 𝑝 p -values that show the statistical importance at p < 0.05 𝑝 0.05 p<0.05 are boldfaced).

Without	Rotate	Flip	Zoom	Mixed
0.005	0.02	0.02	0.05	$>$0.1

Table 6. TABLE VI: Training-time data augmentation applied to the proposed spectral-spatial deep network brings statistically significant improvements in almost all cases (the p 𝑝 p -values that show the statistical importance at p < 0.05 𝑝 0.05 p<0.05 are boldfaced).

	Rotate	Flip	Zoom	Mixed
Without	0.001	$>$0.2	0.05	0.001
Rotate		0.001	0.001	0.05
Flip			$>$0.05	0.001
Zoom				0.001

Full text

I Introduction

Hyperspectral image analysis has been continuously gaining research attention, as the current advances in sensor design have made the acquisition of such imagery much more affordable nowadays. Hyperspectral images (HSI) capture a wide spectrum of light per pixel, and can be represented as large 3D tensors, commonly including hundreds of reflectance values per pixel, characterizing an underlying material. Classification and segmentation (assigning class labels to specific pixels, and finding the boundaries of the objects within the image, respectively) of HSI allow us to build an understanding of the captured scene. Such non-invasive extraction of the spectral footprint of the analyzed objects has multiple real-life applications ranging across various fields, including remote sensing, medicine, biochemistry, and more [1].

In general, the approaches towards automatic HSI classification are divided into conventional image-analysis and machine-learning techniques, and deep learning-powered algorithms. In the former approaches, the input data undergoes the feature engineering process, in which hand-crafted features are extracted and selected [2, 3]. On the other hand, deep learning allows us to intrinsically learn an embedded data representation using the available training data [4, 5, 6, 7, 8, 9, 10]. Deep learning has been tremendously successful in practically all fields related to computer vision and image analysis, and it is the current research mainstream in HSI analysis. Deep networks for HSI classification can be split into three groups, encompassing those operating exclusively on (i) spectral and (ii) spatial pixel characteristics, and (iii) coupling spectral and spatial information for better performance [11, 12].

In [13], the authors compared spectral, spatial, and spectral-spatial deep networks, showing that integrating the spectral and spatial features by kernels with large spectral and small spatial receptive fields helps boost the classification accuracy, if the appropriate regularization is applied (due to the limited size of training sets). Spectral-spatial residual networks employ residual blocks to additionally connect every other 3D convolutional layer, and were shown to be alleviating the declining-accuracy phenomenon of other deep learning models [14]. Hamida et al. [15] generate 3D feature maps using 3D kernels which are gradually turned into 1D feature vectors all along the layers in their 3D network. In [16], the authors enhanced their spectral-spatial networks with a border mirroring strategy to effectively process the areas which are positioned next to the border of an input HSI. An interesting approach of coupling deep learning with multiple feature learning was introduced in [10], where a deep network benefits from the additionally-extracted features. Finally, an end-to-end architecture (called BASS) for efficient band-specific feature learning, while keeping the number of trainable parameters as low as possible, was introduced by Santara et al. [8].

Although acquiring hyperspectral data has been becoming much more affordable, its efficient transfer and analysis is still very time- and cost-inefficient. Therefore, generating new ground-truth datasets is infeasible at a large scale. It is reflected in a very small number of hyperspectral benchmarks which may be exploited to train classification engines in a supervised way [17]. Overall, there are three approaches for dealing with the problem of limited ground-truth HSI. In unsupervised segmentation, prior class labels are not utilized at all, and the input data undergoes clustering into coherent regions of pixels manifesting similar characteristics. The class labels for those regions are, however, unknown, hence they require further post-processing to understand the segmentation results. The unsupervised HSI classification techniques include both standard clustering, performed over the input hyperspectral data or extracted features, and various deep network architectures which perform additional representation learning for more robust classification [18]. In our recent work [19], we introduced an end-to-end approach which combines 3D convolutional autoencoders with a clustering layer for this task.

On the other hand, transfer learning and data augmentation techniques are appropriate in scenarios in which the labeled training data exists and can be used for training or fine-tuning the pre-trained models. Transfer learning is aimed at pre-training a network using a source dataset (as large and representative as possible), and fine-tune it over the target set (often much smaller) which exposes the characteristics of the test data. In contrast, data augmentation techniques either generate additional training data samples that follow the original data distribution and include them in the training set to enhance its representativeness and size, or elaborate new samples during the inference to benefit from an ensemble-like classification engine [20]. Although data augmentation has been shown very effective in allowing large-capacity learners train from small training samples, it has not been intensively applied in HSI analysis, especially in the case of spectral-spatial deep networks (instead, active learning-guided classification model was applied in this context [21]). Here, we shed more light on the training-time data augmentation for spectral-spatial deep convolutional neural networks, and thoroughly verify its impact on the classification abilities of the models, as well as their training and inference times.

In this letter, we introduce a new spectral-spatial convolutional neural network which combines both spectral and spatial hyperspectral pixel features extracted by its initial 3D convolutional layers (Section II). To fully benefit from the local features in the spectral and spatial dimensions (as subtle differences across materials are often manifested in a tiny part of the spectrum, and the pixel sizes can easily become very large, especially in Earth observation scenarios where HSI is acquired by an imaging satellite), our kernels are kept very small and move with unit stride in each dimension to effectively capture fine-grained features. It is in contrast to other state-of-the-art spectral-spatial models which often employ “large” kernels (in the spectral dimension). Our deep network is enhanced by a battery of data augmentation operations applied over 3D hyperspectral patches. A rigorous experimental study performed over the most popular HSI scenes revealed that the proposed approach is competitive against other 3D deep networks from the literature, and offers instant HSI classification (Section III). Also, we analyze the statistical impact of training-time data augmentation on the overall performance of such large-capacity learners.

II Our Method

We introduce a new spectral-spatial convolutional neural network (Fig. 1) which is divided into two parts—the 3D convolutional block, built with a number of 3D convolutional layers, and the classification block performing the final classification of an incoming HSI pixel $h$. For each $h$, we extract a surrounding patch $\mathcal{P}$ of size $(p_{x}\times p_{y}\times p_{w})$, where $p_{x}$, $p_{y}$, and $p_{w}=\mathcal{B}$ represent its width, height, and depth (equal to the number of bands in the input HSI), respectively, with $h$ positioned in its center. To capture fine-grained spectral-spatial features, we apply $\mathcal{K}=24$ kernels of size $(3\times 3\times 3)$ in each layer, and the patch size of $(7\times 7)$, therefore $p_{x}=p_{y}=7$. These kernels move with unit stride in all dimensions. In the feature extraction part of the network, we exploit zero padding and the rectified linear unit (ReLU) as a non-linearity. Therefore, the dimensionality of the input tensor changes in the consecutive layers—let us assume we want to obtain a class label for a hyperspectral pixel represented by $\mathcal{B}=200$ reflectance values (bands). In this scenario, we would obtain the following tensors corresponding to the feature maps throughout the feature extraction process (starting from the input patch $\mathcal{P}$ extracted for this pixel from the input HSI, to the output of the last layer in the 3D convolutional block): $\left[7,7,200\right]\rightarrow\left[5,5,198\right]\rightarrow\left[3,3,196\right]\rightarrow\left[1,1,194\right]$. The number of feature maps at each level remains constant, and equals the number of kernels $\mathcal{K}=24$. In the inner layers, all 3D kernels ($\mathcal{K}=24$) are convolved with each feature map, and the resulting intermediate feature maps ($\mathcal{K}$ for each kernel) are summed together to form the final map for a specific kernel. In the classification part of the model, a dense block with four fully-connected layers (with ReLU activation) containing 512, 256, 128, and $\mathcal{C}$ neurons, where $\mathcal{C}$ denotes the number of classes in the corresponding dataset, is exploited.

Since the capacity of the model becomes very large (it could be decreased by applying a shallower dense block or other supervised learner though), and the network can easily overfit to the training data, we augment the training sets and generate synthetic training patches using rotation, flipping, zooming, and a random mix of all of the aforementioned operations, each applied with the same probability. Since the augmentations are performed at the patch level, we extract larger patches for rotation, to make sure that the rotated patches are not cropped (and to ultimately avoid padding). Such artificial patches are appended to the training set—we at most doubled the number of original samples, unless that number would exceed the most numerous class. In this case, we augment only the missing difference, to ensure that at least half of the training examples are original.

Training of the deep network proceeds until the termination condition has been reached: the maximum execution time has elapsed, the number of epochs has been processed, or an early stopping has been triggered (e.g., the loss value stagnates for a given number of consecutive epochs). In this letter, we combine the early stopping (15 training epochs without improvement in the cross-entropy loss value) with the maximum number of epochs (in all experiments reported in this work, we restricted the number of epochs to 200).

III Experiments

The objectives of our experimental study are two-fold. First, we compare our 3D convolutional neural network with two other spectral-spatial techniques from the literature: the BASS deep network [8], and a method coupling deep networks with multiple feature learning (we refer to it as 3D-CNN) [10]. Second, we investigate the impact of our training-time data augmentation procedures on the abilities of the deep models. For fair comparison, we execute all of our training-time data augmentations for both deep models from the literature too.

In this letter, we exploited three most popular HSI benchmarks111These datasets are available at http://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes. from the literature [17]: (i) Salinas Valley (Sa), USA ($217\times 512$ pixels, acquired by the AVIRIS sensor) presenting different sorts of vegetation (16 classes, 224 bands, with 3.7 m spatial resolution); (ii) Indian Pines (IP), USA ($145\times 145$, AVIRIS) covering the agriculture and forest (16 classes, 200 channels, 20 m); (iii) Pavia University (PU), Italy ($340\times 610$, ROSIS) capturing the urban scenery (9 classes, 103 channels, 1.3 m). We report the average accuracy (AA), overall accuracy (OA), and the kappa scores $\kappa=1-\frac{1-p_{o}}{1-p_{e}}$, where $p_{o}$ and $p_{e}$ are the relative observed agreement, and hypothetical probability of chance agreement, respectively, and $-1\leq\kappa\leq 1$ (the larger $\kappa$ is, the better performance is), obtained over the test sets. To avoid any training-test information leaks, we utilize our challenging patch-based data splits [17], and perform five-fold cross-validation222Our patch-based data splits are available at https://tinyurl.com/ieee-grsl. (we run the algorithms $5\times$ for each fold, thus we have 25 executions of each method altogether). The deep neural network models were implemented in Python 3.6, and were trained using Adam with the learning rate of $10^{-4}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and the batch size of 64.

The experimental results are gathered in Tables I–III for Salinas Valley, Indian Pines, and Pavia University, respectively. We can appreciate that our deep models lead to consistently high-quality classification for all datasets, and significantly outperform BASS for all sets, and 3D-CNN for Salinas Valley and Indian Pines. Importantly, there are only sporadic cases in which our method delivered the globally worst results for specific classes (C1 for Sa, and C2 for PU obtained with our deep network coupled with the mixed and flip augmentation, respectively). It indicates that the proposed architecture deals well with very imbalanced and under-represented classes, and offers accurate classification in such scenarios. Note that for C7 in IP and PU all investigated methods failed, and obtained 0% accuracy—for these classes, the patch-based division contained either no examples in the training or test sets, hence the models were not able to effectively learn those classes (therefore, they could not appropriately cope with unknown examples presented in the test sets of the corresponding folds).

In Fig. 2, we visualize the average kappa scores obtained for all deep models in all investigated augmentation scenarios. The results confirm our previous observations—our spectral-spatial networks obtained better classification when compared with BASS over all training-time augmentations. On the other hand, learning additional features which is a pivotal part of 3D-CNN appeared extremely useful in Pavia University, where 3D-CNN outperformed both BASS and our deep models by a significant margin—see the pair-wise comparison of 3D-CNN with BASS and Ours in Fig. 2c. It indicates that for the sets in which there exist over-represented classes (the Meadows class constitutes more than 40% of all samples) extracting additional features, exploited later by deep models, is extremely beneficial, as the deep nets are unable to capture distinctive features of the minority classes from very limited training samples.

The ranking obtained over the kappa scores, averaged across all hyperspectral scenes (AR) shows that our models delivered better classification results when compared with both BASS and 3D-CNN on average (Table IV). Also, training-time data augmentation brought noticeable improvements in classification accuracy for all networks, with the data augmentation exploiting only rotation being the best for all deep learners.

To verify the statistical importance of the experimental results, we executed two-tailed Wilcoxon tests over the null hypothesis saying that “applying 3D-CNN and our spectral-spatial deep models leads to the same-quality per-class classification” (we focused on these two models as they clearly outperformed the BASS architecture, as presented in Tables I–IV and Fig. 2). The results show that this hypothesis can be safely rejected at $p<0.05$ (Table V)—our deep models delivered statistically better per-class classification accuracies. For details, see the class accuracies for Ours and 3D-CNN reported in Tables I–III—they should be confronted in a pair-wise manner, for all augmentation techniques separately.

In Table VI, we present the $p$-values resulting from the Wilcoxon tests verifying the importance of the training-time data augmentation applied to our deep models. Similarly, extending training sets with synthetically-generated examples leads to statistically different classification performance in almost all cases (the augmentation which involved only flipping is an exception here). Also, zooming and flipping of the the training patches gave the same-quality classification. Overall, the results show that applying data augmentation is a powerful remedy in dealing with small training samples for spectral-spatial deep networks, and can significantly boost the abilities of the underlying (large-capacity) deep learners.

Finally, we analyzed the training and inference times of the investigated techniques in Table IV. Since training-time augmentation increases the size of the training set (which is in contrast to the test-time augmentation [20]), these approaches will always affect the training times of any supervised learner. On the other hand, the time of classifying test examples during the operational phase of a learned model is extremely fast in all methods, and all of them offer real-time operation. Also, it is worth noticing that all of the models can be trained in very affordable time with a single GPU—even in the case of the models introduced in this letter, in which a significant number of small 3D convolutional kernels move in both spectral and spatial dimensions while processing an input HSI.

IV Conclusions

We introduced a new spectral-spatial deep neural network architecture that is composed of the convolutional block which extracts features from the input HSI by applying 3D convolutional layers, and the classification block which performs the final classification. Our rigorous experimental study backed up with statistical tests revealed that the proposed method is competitive with other spectral-spatial architectures from the literature, not only in terms of the classification abilities, but also in terms of training and test time. Although our networks employ small 3D kernels in the convolutional block, the experiments showed that it offers real-time inference. Finally, we showed that applying training-time data augmentation to enhance the representativeness of training sets significantly improves the abilities of underlying deep models.