CoSpace: Common Subspace Learning from Hyperspectral-Multispectral Correspondences

TL;DR

This paper introduces CoSpace, a novel framework for joint subspace learning from hyperspectral and multispectral satellite images, improving land cover classification by leveraging shared spectral information.

Contribution

The paper proposes a new cross-modality feature learning method that aligns hyperspectral and multispectral data in a shared subspace for enhanced classification accuracy.

Findings

CoSpace outperforms existing methods on simulated datasets.

Shared subspace improves spectral information utilization.

Method effectively handles trade-offs between coverage and spectral resolution.

Abstract

With a large amount of open satellite multispectral imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global multispectral land cover classification. However, its limited spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to multispectral ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace), by jointly considering subspace learning and supervised classification. By locally aligning the manifold structure of the two modalities, CoSpace linearly learns a shared latent subspace from hyperspectral-multispectral(HS-MS) correspondences. The multispectral out-of-samples can be then projected into the subspace, which are expected to take advantages of rich spectral information of the corresponding hyperspectral data used for learning, and thus leads to a better classification. Extensive experiments on two simulated HSMS datasets (University of Houston and Chikusei), where HS-MS data sets have trade-offs between coverage and spectral resolution, are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.

Tables

Table 1. TABLE I: The number of training and testing samples for the University of Houston MS-HS dataset.

Class No.	Class Name	Training	Testing
1	Healthy Grass	537	699
2	Stressed Grass	61	1154
3	Synthetic Grass	340	357
4	Tree	209	1035
5	Soil	74	1168
6	Water	22	303
7	Residential	52	1203
8	Commercial	320	924
9	Road	76	1149
10	Highway	279	948
11	Railway	33	1185
12	Parking Lot1	329	904
13	Parking Lot2	20	449
14	Tennis Court	266	162
15	Running Track	279	381
	Total	2897	12021

Table 2. TABLE III: Quantitative performance comparison with the different algorithms on the University of Houston data. The best one is shown in bold.

Algorithm	Baseline (%)			P-JDR (%)			L-USMA (%)			L-SMA (%)			CoSpace (%)
Parameter	/			$d=30$			$(k,\sigma ,d)=(10,1,20)$			$d=30$			$(\alpha ,\beta ,d)=(0.01,0.01,30)$
Classifier	1NN	LSVM	CCF	1NN	LSVM	CCF	1NN	LSVM	CCF	1NN	LSVM	CCF	1NN	LSVM	CCF
OA	61.70	62.12	68.21	66.37	64.31	67.13	66.33	65.54	68.41	65.04	68.01	69.59	68.82	69.38	72.17
AA	65.57	65.97	70.47	69.23	67.50	69.64	69.37	68.81	70.84	68.15	70.50	71.02	71.29	71.69	73.56
$\kappa$	0.5842	0.5889	0.6543	0.6345	0.6118	0.6430	0.6342	0.6251	0.6565	0.6202	0.6520	0.6695	0.6613	0.6672	0.6975
Class1	69.10	76.39	67.95	73.39	71.82	70.24	80.40	78.68	67.67	81.69	75.25	68.53	88.56	75.54	69.96
Class2	79.20	80.59	78.08	81.20	81.72	72.53	78.94	79.90	75.04	90.99	97.57	77.90	73.48	73.74	77.99
Class3	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Class4	85.22	85.51	92.27	87.63	91.21	93.24	94.30	96.23	92.85	88.60	94.78	98.74	93.62	98.74	98.26
Class5	98.89	99.06	99.40	98.72	98.72	99.32	99.49	99.40	98.89	99.40	98.97	99.14	98.97	99.40	99.40
Class6	86.14	86.14	86.14	86.47	86.14	76.57	86.14	86.47	86.47	86.47	86.47	70.96	85.15	85.48	85.15
Class7	36.49	50.62	63.76	59.6	53.78	51.04	49.79	50.21	63.18	58.19	72.32	77.14	69.99	73.98	80.05
Class8	50.76	56.49	56.06	59.42	59.42	59.09	54.76	66.23	56.82	55.74	62.01	62.23	58.77	63.53	62.01
Class9	60.92	56.22	70.58	64.14	63.01	72.32	63.45	65.19	69.10	47.17	49.96	61.27	64.40	59.79	64.93
Class10	40.93	45.36	45.25	44.41	49.16	45.36	44.09	53.90	47.47	49.47	58.12	52.32	45.15	64.14	57.70
Class11	39.16	27.43	43.88	36.03	35.53	39.41	45.91	29.79	43.71	35.36	28.86	36.46	44.14	36.54	47.26
Class12	41.37	31.64	56.08	51.55	25.66	65.82	45.8	29.76	62.06	65.04	35.84	62.5	50.22	46.79	62.72
Class13	0.45	0.00	0.67	0.67	0.00	2.45	0.22	0.00	1.11	4.68	0.00	0.00	0.89	0.00	0.45
Class14	97.53	97.53	98.77	98.15	98.77	99.38	99.38	98.77	100.00	97.53	100.00	100.00	98.15	100.00	99.38
Class15	97.38	96.59	98.16	97.11	97.64	97.9	97.9	97.64	98.16	97.9	97.38	98.16	97.9	97.64	98.16

Table 3. TABLE IV: Quantitative performance comparison with the different algorithms on the Chikusei data. The best one is shown in bold.

Algorithm	Baseline (%)			P-JDR (%)			L-USMA (%)			L-SMA (%)			CoSpace (%)
Parameter	/			$d=30$			$(k,\sigma ,d)=(10,1,30)$			$d=20$			$(\alpha ,\beta ,d)=(0.1,0.01,30)$
Classifier	1NN	LSVM	CCF	1NN	LSVM	CCF	1NN	LSVM	CCF	1NN	LSVM	CCF	1NN	LSVM	CCF
OA	55.99	60.20	71.11	60.45	68.19	71.87	61.48	67.31	72.33	62.44	67.90	71.53	64.07	71.12	75.69
AA	62.39	69.42	70.40	64.95	71.71	71.02	64.95	70.06	71.49	64.52	70.79	66.47	68.05	73.96	71.46
$\kappa$	0.5084	0.5523	0.6761	0.5614	0.6414	0.6834	0.5715	0.6318	0.6893	0.5812	0.6391	0.6802	0.5995	0.6746	0.7260
Class1	79.49	78.21	80.54	97.2	98.25	81.93	98.48	99.18	81.00	80.89	98.72	82.52	80.19	92.54	79.25
Class2	95.02	94.43	82.7	93.2	93.95	94.86	93.25	93.68	93.95	92.98	93.20	92.50	95.02	93.47	94.91
Class3	11.37	23.54	50.06	14.40	16.03	24.13	31.07	39.37	56.74	61.81	62.57	55.31	80.10	80.40	77.71
Class4	91.64	92.13	92.56	88.65	90.87	93.38	89.49	92.21	94.90	86.54	90.57	91.53	93.47	90.59	96.23
Class5	87.00	97.65	94.68	67.78	70.94	76.26	63.90	34.57	79.87	25.36	28.43	16.06	74.28	83.94	66.52
Class6	65.36	62.01	81.48	55.18	75.11	87.77	50.90	67.15	79.42	48.29	62.52	78.91	53.49	63.61	79.02
Class7	99.26	99.67	99.93	91.95	98.43	99.98	91.02	97.00	99.71	95.46	96.87	97.79	88.74	97.74	99.75
Class8	43.42	57.11	93.40	85.32	95.58	97.95	91.64	96.32	99.29	93.93	95.59	93.49	65.61	95.05	92.72
Class9	99.92	100.00	100.00	98.34	98.66	99.53	98.82	99.45	99.92	99.21	99.53	99.13	100.00	98.66	99.76
Class10	16.96	24.81	19.56	21.36	23.75	22.24	19.78	20.13	17.16	21.15	21.39	15.48	22.68	22.35	18.00
Class11	2.11	0.00	2.11	0.63	1.05	6.53	0.00	0.00	3.37	0.00	0.00	0.00	0.00	0.00	0.00
Class12	87.85	90.32	88.91	86.44	88.20	89.96	81.51	88.38	87.32	84.43	90.14	85.92	86.62	90.32	80.46
Class13	33.08	33.11	33.09	33.08	33.11	33.11	32.36	33.01	33.11	25.47	32.61	56.25	33.11	33.11	67.90
Class14	67.75	94.20	85.38	59.40	58.24	99.77	58.70	59.40	89.56	71.93	72.85	59.40	59.40	59.40	52.44
Class15	97.33	100.00	100.00	100.00	100.00	100.00	96.26	96.79	100.00	94.12	93.58	100.00	100.00	93.58	97.86
Class16	66.24	74.88	88.62	95.39	95.58	99.71	99.80	98.43	100.00	99.61	99.71	99.51	93.13	97.84	100.00
Class17	16.79	58.03	3.84	15.83	81.29	0.24	7.19	76.26	0.00	15.59	65.23	7.91	30.94	64.75	0.00

Equations

$$\tilde{\mathbf{Y}}=\mathbf{P\Theta }\tilde{\mathbf{X}}+\mathbf{E},$$

$$\mathbf{J}=$$

$$\mathbf{\Theta }=$$

$$\times (\mu \tilde{\mathbf{X}}{\tilde{\mathbf{X}}}^T+\mu \mathbf{I}+\beta \tilde{\mathbf{X}}\mathbf{L}{\tilde{\mathbf{X}}}^T{)}^{-1}.$$

$$\left[\mathbf{U},\mathbf{S},\mathbf{V}\right]=\mathrm{svd}(\mathbf{\Theta }-{\mathbf{\Lambda }}_2/\mu ),\mathbf{G}=\mathbf{U}{\mathbf{I}}_{n\times m}\mathbf{V}.$$

$${\mathbf{\Lambda }}_1\leftarrow {\mathbf{\Lambda }}_1+\mu (\mathbf{J}-\mathbf{\Theta }\tilde{\mathbf{X}}),{\mathbf{\Lambda }}_2\leftarrow {\mathbf{\Lambda }}_2+\mu (\mathbf{G}-\mathbf{\Theta }).$$

$$\mu =\min (\rho \mu ,{\mu }_{\max }).$$

$${\min }_{\mathbf{P},\mathbf{\Theta }}\left\{\begin{array}{rl}\frac{1}{2}\Vert \tilde{\mathbf{Y}}-& \mathbf{P\Theta }\tilde{\mathbf{X}}{\Vert }_F^2+\Phi (\mathbf{P})+\Psi (\mathbf{\Theta })\\ & s.t.\mathbf{\Theta }{\mathbf{\Theta }}^T=\mathbf{I}\end{array}\right\}.$$

$$\Phi (\mathbf{P})=\frac{\alpha }{2}\Vert \mathbf{P}{\Vert }_F^2,$$

$$\Psi (\mathbf{\Theta })=\frac{\beta }{2}\mathrm{tr}(\mathbf{\Theta }\tilde{\mathbf{X}}\mathbf{L}(\mathbf{\Theta }\tilde{\mathbf{X}}{)}^T),$$

$${\mathbf{W}}_{i,j}=\{\begin{array}{l}\begin{array}{rl}1/N_k,& \text{if Xi and Xj belong to the k-th class;}\\ 0,& \text{otherwise,}\end{array}\end{array}$$

$${\min }_{\mathbf{P}}\left\{\begin{array}{r}\frac{1}{2}\Vert \tilde{\mathbf{Y}}-\mathbf{P\Theta }\tilde{\mathbf{X}}{\Vert }_F^2+\frac{\alpha }{2}\Vert \mathbf{P}{\Vert }_F^2\end{array}\right\},$$

$$\mathbf{P}=(\tilde{\mathbf{Y}}{\mathbf{Q}}^T)(\mathbf{Q}{\mathbf{Q}}^T+\alpha \mathbf{I}{)}^{-1},$$

$${\min }_{\mathbf{\Theta }}\left\{\begin{array}{rl}\frac{1}{2}\Vert \tilde{\mathbf{Y}}-\mathbf{P}& \mathbf{\Theta }\tilde{\mathbf{X}}{\Vert }_F^2+\frac{\beta }{2}\mathrm{tr}(\mathbf{\Theta }\tilde{\mathbf{X}}\mathbf{L}(\mathbf{\Theta }\tilde{\mathbf{X}}{)}^T)\\ & s.t.\mathbf{\Theta }{\mathbf{\Theta }}^T=\mathbf{I}\end{array}\right\}.$$

$${\min }_{\mathbf{\Theta },\mathbf{J},\mathbf{G}}\left\{\begin{array}{rl}& \frac{1}{2}\Vert \tilde{\mathbf{Y}}-\mathbf{PJ}{\Vert }_F^2+\frac{\beta }{2}\mathrm{tr}(\mathbf{\Theta }\tilde{\mathbf{X}}\mathbf{L}(\mathbf{\Theta }\tilde{\mathbf{X}}{)}^T)\\ & s.t.\mathbf{J}=\mathbf{\Theta }\tilde{\mathbf{X}},\mathbf{G}=\mathbf{\Theta },\mathbf{G}{\mathbf{G}}^T=\mathbf{I}\end{array}\right\}.$$

$${\mathcal{L}}_C\left(\mathbf{\Theta },\mathbf{J},\mathbf{G},{\mathbf{\Lambda }}_1,{\mathbf{\Lambda }}_2\right)$$

$$=$$

$$+$$

$$s.t.\mathbf{G}{\mathbf{G}}^T=\mathbf{I},$$

$${\min }_{\mathbf{\Theta }}\left\{\begin{array}{rl}& \frac{\beta }{2}\mathrm{tr}(\mathbf{\Theta }\tilde{\mathbf{X}}\mathbf{L}(\mathbf{\Theta }\tilde{\mathbf{X}}{)}^T)+{\mathbf{\Lambda }}_1^T(\mathbf{J}-\mathbf{\Theta }\tilde{\mathbf{X}})\\ +{\mathbf{\Lambda }}_2^T(& \mathbf{G}-\mathbf{\Theta })+\frac{\mu }{2}\Vert \mathbf{J}-\mathbf{\Theta }\tilde{\mathbf{X}}{\Vert }_F^2+\frac{\mu }{2}\Vert \mathbf{G}-\mathbf{\Theta }{\Vert }_F^2\}\end{array}\right\},$$

$$\mathbf{\Theta }\leftarrow$$

$$\times (\mu \tilde{\mathbf{X}}{\tilde{\mathbf{X}}}^T+\mu \mathbf{I}+\beta \tilde{\mathbf{X}}\mathbf{L}{\tilde{\mathbf{X}}}^T{)}^{-1}.$$

$${\min }_{\mathbf{J}}\left\{\begin{array}{r}\frac{1}{2}\Vert \tilde{\mathbf{Y}}-\mathbf{PJ}{\Vert }_F^2+{\mathbf{\Lambda }}_1^T(\mathbf{J}-\mathbf{\Theta }\tilde{\mathbf{X}})+\frac{\mu }{2}\Vert \mathbf{J}-\mathbf{\Theta }\tilde{\mathbf{X}}{\Vert }_F^2\end{array}\right\},$$

$$\mathbf{J}\leftarrow ({\mathbf{P}}^T\mathbf{P}+\mu \mathbf{I}{)}^{-1}({\mathbf{P}}^T\tilde{\mathbf{Y}}+\mu \mathbf{\Theta }\tilde{\mathbf{X}}-{\mathbf{\Lambda }}_1).$$

$${\min }_{\mathbf{G}}\left\{\begin{array}{r}{\mathbf{\Lambda }}_2^T(\mathbf{G}-\mathbf{\Theta })+\frac{\mu }{2}\Vert \mathbf{G}-\mathbf{\Theta }{\Vert }_F^2\},s.t.\mathbf{G}{\mathbf{G}}^T=\mathbf{I}\end{array}\right\},$$

$$\left[\mathbf{U},\mathbf{S},\mathbf{V}\right]=\mathrm{svd}(\mathbf{\Theta }-{\mathbf{\Lambda }}_2/\mu ),$$

$$\mathbf{G}\leftarrow \mathbf{U}{\mathbf{I}}_{n\times m}\mathbf{V}.$$

$${\mathbf{\Lambda }}_1\leftarrow {\mathbf{\Lambda }}_1+\mu (\mathbf{J}-\mathbf{\Theta }\tilde{\mathbf{X}}),{\mathbf{\Lambda }}_2\leftarrow {\mathbf{\Lambda }}_2+\mu (\mathbf{G}-\mathbf{\Theta }),$$

$$\mu \leftarrow \min (\rho \mu ,{\mu }_{\max }).$$

Full text

I Introduction

Recently, the launch of operational optical broadband (multispectral) satellites has successfully boosted the usage of multispectral data for various tasks such as urban monitoring, management of natural resources, ecosystem, and disasters prediction. There has been a growing interest in large-scale land cover mapping of urban [1], agriculture monitoring [2][3], and mineral exploration [4], since high-quality multispectral (MS) satellite imagery is openly available on a global scale (e.g., Sentinel-2 and Landsat-8). However, MS data fails to discriminate spectrally similar classes due to its broad spectral bandwidth. Hyperspectral imaging can acquire richer spectral information that enables high discrimination ability but its coverage from space is much narrower than the one of MS imaging due to the limitations of imaging devices and satellite techniques. This trade-off naturally motivates us to ponder a question: can HS imagery covering only a limited part of the MS imagery be explored to improve the classification of the entire area covered by the MS imagery? This is as a typical cross-modal feature learning problem.

Researchers have proposed a variety of multi-modal feature learning algorithms by introducing additional information, which can be roughly categorized into two parts: fusion-based joint feature learning (FJFL) [5, 6] and alignment-based shared feature learning (ASFL) [7]. The main difference between FJFL and ASFL is illustrated in Fig. 1.

FJFL aims to learn discriminative features by absorbing the different properties from multi-modal data. FJFL fuses the different sources at the data level to diversify the information and then to further learn the higher-level feature representation. One intuitive way for FJFL is to directly learn a joint data representation at the feature level. At present, this is the mainstream approach for multi-modal data analysis [8]. For example, by embedding the height information from LiDAR into MS (HS) data, Ghamisi et al. [9] learned multi-fold features from HS and LiDAR correspondences for a multi-modal classification task. Iyer et al. [10] provided a graph-based new perspective for feature extraction and segmentation of multi-modal images and achieved a desirable result. The resulting discriminative features are beneficial for improving the performance of some high-level applications, especially classification [11][12], object detection [13], image/video analysis [14], and spectral unmixing [15]. Image fusion can be also regarded as a part of FJFL when feature learning is applied subsequently. For instance, hyperspectral and multispectral (HS-MS) data fusion enhances the spectral resolution of MS data by fusing it with the low-spatial-resolution HS data [5]. The fused HS-MS product can be then seen as a new input for further discriminative feature learning.

Behind the advancement of FJFL, the complete data correspondence is the prerequisite. This limitation undoubtedly results in a poor fit for cross-modal data analysis, in particular for cross-modal feature learning [16]111In contrast to multi-modal learning (bi-modality for example), cross-modal learning trains on single modality and tests on bi-modality, or *vice versa *(train on bi-modality and test on single modality).. In our MS-HS case, the cross-modal learning refers to a problem that given a large-scale MS image and a limited HS area partially overlapping with the MS data (see Fig. 2 for example), we learn the low-dimensional embedding representation from the limited amount of MS-HS correspondences and transfer the learned features to the rest of MS data for improving the performance of large-scale land-cover and land-use mapping. During the process, we expect to transfer the discrimination capability learnt from the rich spectral information into MS data through the learned common subspace in order to more effectively identify some challenging classes that are hardly recognized by MS data due to its poor spectral information. Please note that we just start a preliminary investigation of cross-modal learning (MS-HS) in this paper, that is, the MS and HS images share the same land-cover classes.

Unlike FJFL, ASFL is more apt for cross-modal feature learning, since ASFL can adaptively shuttle back and forth between the different modalities or domains by means of the learned common subspace. Matasci et al. [17] linearly projected the hyperspectral data of the source and target domains into a common feature space where the gap between domains in hyperspectral image classification are expected to be reduced. Kulis et al. [18] addressed the issue of visual domain adaption by learning a nonlinear transformation in kernel space, with the application to general object recognition. In [19], a probabilistic framework was proposed to align the class distributions of two domains for robust hyperspectral image classification. Manifold alignment (MA) [20] is also a powerful tool for modeling this kind of issue. Inspired by MA, Tuia et al. [7] aligned multi-view remote sensing images on manifolds by fully allowing for the spectral variabilities between the different angle imageries, yielding a significant improvement of classification performance.

It should be noted that these methods mentioned above only consider the differences of a unimodality between the source and target domains at the level of original features, but they fail to investigate the transferability of multi-modality since the different modalities usually hold the different feature dimensions. Although these approaches can build connections between features or instances, a poorly connected relationship between the learned common subspace and label information is still hindering the low-dimensional feature representation from being more discriminative.

We propose a cross-modality feature learning framework, called common subspace learning (CoSpace), that learns the shared feature representation (common subspace) from partial HS-MS correspondences. Extensive experiments are conducted on simulated MS and partially overlapped real HS data based on two airborne HS datasets: the University of Houston and Chikusei datasets. MS data is generated from HS data by using the spectral response functions (SRFs) of Sentinel-2. We re-label the training and testing classes on the datasets to meet the problem setting of cross-modal feature learning and further to make them more challenging (see Section III for details). Our contributions can be specifically unfolded as follows:

• •

We propose a novel common subspace learning (CoSpace) approach by jointly considering the subspace learning and classification in order to effectively bridge the learned features and label information, aiming at addressing the HS-MS cross-modal feature learning issue;

• •

By locally aligning HS-MS data on the low-dimensional manifolds where the features of HS and MS share the same dimension, CoSpace linearly learns a latent shared subspace from HS-MS correspondences, where samples are expected to be better classified. Owing to the subspace learned in a linear way, the out-of-samples data can be simply and smoothly embedded;

• •

An optimization algorithm based on the alternating direction method of multipliers (ADMM) is designed to solve the proposed model.

The remainder of this paper is organized as follows. In section II, we first clarify our motivation and then propose the methodology of the CoSpace model, finally elaborate on the corresponding ADMM-based optimization algorithm. Section III presents the experimental results and analysis on two different HS-MS datasets both qualitatively and quantitatively. Finally, some conclusions are drawn in Section IV.

II CoSpace: Common Subspace Learning

To take the benefit of HS imagery covering only a limited part of the MS imagery, and subsequently improve the classification results of the entire area covered by the MS imagery, our idea is to learn a HS-MS common subspace, in which the data from one domain can be adaptively transferred to another domain. Fig. 2 shows the holistic diagram of the proposed CoSpace method.

II-A Problem Formulation

Let $\mathbf{X}_{M}\in\mathbb{R}^{d_{M}\times N}$ and $\mathbf{X}_{H}\in\mathbb{R}^{d_{H}\times N}$ be the observed MS image with $d_{M}$ bands by $N$ pixels and the HS image with $d_{H}$ bands by $N$ pixels, respectively. $\mathbf{Y}\in\mathbb{R}^{L\times N}$ is the label matrix represented by one-hot encoding. $\mathbf{\Theta}_{M}\in\mathbb{R}^{d\times d_{M}}$ ($\mathbf{\Theta}_{H}\in\mathbb{R}^{d\times d_{H}}$) is denoted as the projection matrix for connecting the MS (HS) data and the latent subspace. The variable $\mathbf{P}\in\mathbb{R}^{L\times d}$ is the weighted matrix specified by bridging the latent subspace and label information. Accordingly, $\mathbf{\widetilde{Y}}=[\mathbf{Y},\mathbf{Y}]\in\mathbb{R}^{L\times 2N}$ can be modeled as follows:

[TABLE]

where $\widetilde{\mathbf{X}}=\begin{bmatrix}\mathbf{X}_{M}&\mathbf{0}\\ \mathbf{0}&\mathbf{X}_{H}\end{bmatrix}\in\mathbb{R}^{(d_{M}+d_{H})\times 2N}$, and $\mathbf{\Theta}=[\mathbf{\Theta}_{M},\mathbf{\Theta}_{H}]\in\mathbb{R}^{d\times(d_{M}+d_{H})}$. $\mathbf{E}\in\mathbb{R}^{L\times 2N}$ is the corresponding residual matrix containing the additive noise and other errors.

Since the Eq. (1) is a typically ill-posed problem owing to more degrees of flexibility involved (e.g. latent subspace estimation), several assumptions (or prior knowledge) should be introduced into CoSpace using regularization technique. Followed by a popular joint learning framework proposed in [21], we formulate the CoSpace as the following constrained optimization problem:

[TABLE]

The two regularization terms in Eq. (2) are detailed in the following:

To achieve a reliable generalization of our model, the variable $\mathbf{P}$ parameterized by $\alpha$ can be regularized by a Frobenius Norm:

[TABLE]

and the prior knowledge with respect to $\mathbf{\Theta}$, resulting in a multi-modal manifold alignment regularization, can be expressed with a joint graph structure as

[TABLE]

where $\mathbf{L}=\mathbf{D}-\mathbf{W}\in\mathbb{R}^{2N\times 2N}$ stands for a joint Laplacian matrix, $\mathbf{W}$ that is a corresponding adjacency matrix can be directly inferred from label information in the form of the LDA-like graph [22]:

[TABLE]

and then $\mathbf{D}$ is computed by $\mathbf{D}_{ii}=\sum_{i\neq j}\mathbf{W}_{i,j}$.

II-B Model Optimization

Considering the nonconvexity of problem (2), an iterative alternating optimization strategy is adopted to solve the convex subproblems of each variable $\mathbf{P}$ and $\mathbf{\Theta}$. An implementation of CoSpace is given in Algorithm 1.

Optimization with respect to $\mathbf{P}$: This is a typical least-squares problem with Tikhonov regularization that can be formulated as

[TABLE]

which has a closed-form solution

[TABLE]

where $\mathbf{Q}=\mathbf{\Theta}\widetilde{\mathbf{X}}$.

Optimization with respect to $\mathbf{\Theta}$: the optimization problem for $\mathbf{\Theta}$ can be formulated as

[TABLE]

In order to solve (8) effectively with ADMM, we consider an equivalent form by introducing auxiliary variables $\mathbf{J}$ and $\mathbf{G}$ to replace $\mathbf{\Theta}\widetilde{\mathbf{X}}$ and $\mathbf{\Theta}$, respectively.

[TABLE]

The augmented Lagrangian version of Eq. (9) is

[TABLE]

where $\mathbf{{\Lambda}_{1}}$ and $\mathbf{{\Lambda}_{2}}$ are Lagrange multipliers and $\mu$ is the penalty parameter. Algorithm 2 summarizes the specific procedures for solving the problem (9), and the solution to each subproblem is detailed in Appendix A.

Finally, we repeat these optimization procedures until a stopping criterion is satisfied.

II-C Convergence Analysis

The iterative alternating strategy used in Algorithm 1 and Algorithm 2 is a block coordinate descent (BCD), whose convergence is theoretically guaranteed as long as each subproblem of Eq. (2) is strictly convex, which can be exactly minimized [23]. Moreover, we experimentally display an illustration to clarify the convergence of CoSpace on both HS-MS datasets, where the objective function value is recorded in each iteration (see Fig. 4).

III Experiments

In this section, we quantitatively and qualitatively evaluate the performance of the proposed method on two HS-MS datasets taken over the University of Houston and Chikusei. To validate the transferability of learned features by our CoSpace method, classification is explored as a potential application. Therefore, three different classifiers, namely the nearest neighbor (NN) based on the Euclidean distance, linear support vector machines (LSVM), and canonical correlation forest (CCF) [24] are selected for this task. As a variant of random forest [25], CCF has shown its effectiveness in various tasks [26, 27, 28] owing to supervised feature extraction via canonical correlation analysis when constructing each decision tree. Furthermore, We compare the proposed method (CoSpace) with several classical approaches, which are suitable for the cross-modal feature learning task, including principle component analysis (PCA) based on joint dimensionality reduction (P-JDR for short) [29], locality preserving projection (LPP) based on unsupervised manifold alignment (L-USMA for short) [30], and LPP-based supervised manifold alignment (L-SMA) [31] as well as the original MS (Baseline).

III-A University of Houston HS-MS Datasets

III-A1 Data Description

The HS data was acquired by the ITRES CASI-1500 sensor over an urban area around the campus of the University of Houston, Houston, USA, which was provided in the $2013$ IEEE GRSS data fusion contest [32]. The image consists of $349\times 1905$ pixels with 144 spectral bands in the wavelength from 364 nm to 1046 nm with spectral resolution of 10 nm at a ground sampling distance (GDS) of 2.5 m. Spectral simulation is performed to generate the MS image by degrading the full HS image in the spectral domain using the MS spectral response functions (SRFs) of Sentinel-2 as filters (for more details refer to [5]). Following this, the MS data with dimensions of $349\times 1905\times 10$ is generated. The MS image and the corresponding partial HS image over the University of Houston scene are shown in Fig. 5(a).

III-A2 Experimental Setup

Initially, we re-distribute the training and testing samples, as shown in Fig. 5(a) and more specifically listed in Table II, to meet our problem setting that there is a large amount of the MS data (complete low-quality data) together with a limited amount of the HS data (incomplete high-quality data).

For the performance assessment of the algorithms, we adopt three criteria to quantify experiential results as

1. Overall Accuracy (OA): This index is defined by the ratio between the number of multispectral samples that are correctly classified and the number of corresponding test samples.

2. Average Accuracy (AA): We collect the classification accuracy of each class and average them to achieve an AA-based evaluation.

3. Kappa Coefficient ($\kappa$): It statistically measures the agreement between the final classification map and the ground-truth map. Generally speaking, $\kappa$ is more robust and convincing than a simple percent-based agreement calculation (e.g. OA and AA), since the agreement occurring by chance is fully considered.

Furthermore, we experimentally maximize the performance of the different algorithms by tuning their parameters, such as dimension ($d$), regularization parameters ($\alpha,\beta,\gamma$), etc., using 10-fold cross-validation on training data. For the dimension ($d$) which is a common parameter for all algorithms, they can be selected ranging from $10$ to $50$ at an interval of $10$. For the number of nearest neighbors ($k$) and the standard deviation of Gaussian kernel function ($\sigma$) in L-USMA, we select them in the range of $\{10,20,...,50\}$ and $\{10^{-2},10^{-1},\allowbreak 10^{0},10^{1},10^{2}\}$, respectively, and two regularization parameters ($\alpha,\beta$) in CoSpace are both chosen from $\{10^{-2},10^{-1},\allowbreak 10^{0},10^{1},10^{2}\}$.

III-A3 Results and Analysis

Fig.6 shows the classification maps of compared algorithms using three different classifiers, while Table III details the quantitative assessment results under the optimal parameters determined by cross-validation.

Overall, after absorbing partial HS information, those ASFL approaches are prone to obtain a better classification result, compared to the baseline (only multispectral data). P-JDR steadily outperforms the baseline, especially using 1NN and LSVM classifiers, although its classification accuracy using CCF is slightly lower than baseline’s. By embedding local topological structure of data, L-USMA performs better than baseline, and even P-JDR, showing stable results for three kinds of classifiers. With a more discriminative supervised information, L-SMA obtains more competitive results by locally constructing LDA-like graph, whose performance is basically superior to that of the baseline, P-JDR, and L-USMA. Unlike L-SMA that only aligns different modalities on a common subspace, the proposed CoSpace learns a latent subspace by aligning different modalities but also bridges the learned subspace with label information, achieving a best classification accuracy. Compared to baseline, P-JDR, L-USMA, and L-SMA, CoSpace increases the OAs of $7.12\%$, $2.45\%$, $2.49\%$ , and $3.78\%$, respectively, with 1NN classifier, and $7.26\%$, $5.07\%$, $3.84\%$, and $1.37\%$, respectively, with LSVM classifier, as well as $3.96\%$, $5.04\%$, $3.75\%$, and $2.58\%$, respectively, with CCF classifier. Likewise, there are similar trends for the other indices of AA and $\kappa$, which indicates that CoSpace tends to learn semantically meaningful features.

We can also observe from Fig. 5(a) and Table II that the training samples collected in a very limited area badly results in the data unbalance between different classes. For instance, the number of training samples in Health Grass is dozens of times as much as that in Water, Railway, Residential, Commercial and Parking Lot2. This might make the classifier impossible to be trained effectively, since more attentions are paid on those classes with large-size samples, and, contrariwise, the small-scale classes play relatively less and even nothing.

A feasible solution to the problem is to enhance data representative ability by jointly feature learning from multi-modalities. For the performance evaluation of classifying those small-scale classes, e.g. Residential, Commercial, and Railway, a direct evidence has been shown in Table III that those ASFL-based approaches (e.g. P-JDR, L-USMA, and L-SMA as well as CoSpace) obviously perform better on these small-scale samples than directly using original MS data (baseline). As expected, CoSpace dramatically outperforms the others, particularly on Residential and Railway. There is no denying, however, that CoSpace is superior to other algorithms to a larger extent, although it fails to effectively identify Parking Lot2 as same with others.

To visually highlight the classification differences for the different methods, we enlarge the classification maps of a sub-area overshadowed by the cloud, as shown in Fig. 6 where we can see that the methods with considering the hyperspectral information are able to generate the more discriminative features than the baseline, while the proposed CoSpace yields a better performance in identifying the materials in the shadow area, particularly for vegetation (e.g., Grass), Residential and Commercial that are easily misclassified by the traditional methods.

III-A4 Sensitivity Analysis to the Training Set Size

As the performance of the CoSpace largely depends on the number of training samples, it is, therefore, indispensable to investigate the sensitivity of the training set size. In detail, we conduct the classification using the CoSpace by fixing the test set and setting a series of new training sets randomly selected from the original training set with the different percentages ranging from 5% to 100% at a 5% interval. As can be seen in Fig. 7(a), there is a similar trend in OAs using different classifiers, that is, the classification accuracy improves with the training set size, faster in the early, and later basically stabilized.

III-B Chikusei HS-MS Datasets

III-B1 Data Description

The Headwall’s Hyperspectral Visible and Near-Infrared series C (VNIR-C) imaging sensor acquired the airborne HS dataset over the agricultural and urban areas of Chikusei, Ibaraki, Japan, in 2014. This VNIR-C sensor collected 128 bands covering the wavelength range from 363 nm to 1018 nm with spectral resolution of 10 nm, and the scene consists of $2517\times 2335$ pixels at GSD of 2.5 m. The data set was made available to the scientific community recently, and more details regarding the data acquisition and processing can be found in [33]. Similarly, the MS image with the size of $2517\times 2335\times 10$ was simulated by spectrally down-sampling the full HS image using the known SRFs of Sentinel-2. The generated MS image and the partial HS image over the Chikusei scene are shown in Fig. 5(b).

III-B2 Experimental Setup

Fig. 5(b) shows the latest training and testing labeling of the Chikusei dataset, which is quantified in Table II. Three indices: OA, AA, and $\kappa$ introduced above are calculated to quantitatively assess the classification performance. Similarly to the case of the University of Houston dataset, the parameters for those given algorithms are determined by the 10-fold cross-validation on the training samples and the same range setting with those used for the University of Houston dataset is also conducted to the Chikusei dataset.

III-B3 Results and Analysis

Similar to the University of Houston scene, we evaluate the performance for the Chikusei data both quantitatively and visually. Three classification indices with optimal parameters for different algorithms are summarized in Table IV. For visual comparison, we give the corresponding classification maps in the full scene with those comparative algorithms under the different classifiers, as shown in Fig. 8.

As the classes in the Chikusei scene are more challenging classes and the distribution of training samples is inhomogeneous, directly using original MS data as input fails to identify certain materials, such as Forest, Manmade (Dark),* and Manmade (Grass)*, yielding a poor performance in OA, AA, and $\kappa$. Especially while using 1NN classifier, P-JDR and L-USMA, which belong to the unsupervised feature learning method, observably exceed basline in classification accuracy by $4.46\%$ and $5.49\%$, respectively. For LSVM and CCF classifiers, a similar trend is also demonstrated in Table IV. Owing to the limited training samples and their distribution unbalance, the subspace projection learned by L-SMA easily traps into over-fitting, despite of only having a weak performance improvement compared to these previously compared algorithms. By jointly performing subspace learning and classification, CoSpace not only aligns the different modalities in a latent common subspace, but also connects the subspace with label information formulated by training data. As a result, CoSpace obtains a higher classification accuracy than other algorithms, as listed in Table IV. This might attribute to the learned common subspace, since the features projected in the subspace can absorb various properties from different modalities.

Similarly, we also make a visual comparison by giving a salient region in which the CoSpace’s superiority in classifying complex and similar land-cover classes is further shown as detailed in Fig. 8. Compared to other alignment-based methods, CoSpace is capable of better transferring HS information into MS data by means of joint subspace learning and classification, yielding a more discriminative low-dimensional embedding. The learned features can recognize those classes of holding very similar features in MS data, such as Bare Soil (Farmland) and Row Crops, Weeds in Farmland and Rice Field (Grown), more effectively. As shown in Fig. 8, CoSpace performs more reasonable and competitive classification results, that is, on one hand the Weeds in Farmland and Rice Field (Grown) are most likely to be coexisted in a scene; on the other hand, the Bare Soil (Farmland) and Row Crops are separated more correctly. This can be explained by a powerful transferability of HS information in the proposed CoSpace.

III-B4 Sensitivity Analysis to the Training Set Size

Similar to the MS-HS Houston datasets, we apply the same investigating strategy and observe the trend of classification performance using CoSpace with different sizes of training sets on the MS-HS Chikusei datasets in Fig. 7(b). There is a very substantial change in classification accuracy with the increase of the training set size ranging from 5% to 40% of total training samples, while the performance tends to be stable after the training set size is over 50%.

IV Conclusion

The trade-off between MS and HS imaging in terms of observation ranges and spectral resolution motivates us to ponder whether HS data partially overlapping MS data can contribute to improving the classification performance of the whole MS imagery. For this purpose, we proposed CoSpace to achieve the property transferring in the different domains by learning a latent common subspace. Moreover, an effective joint strategy that simultaneously considers subspace learning and classification is embedded into the proposed method to tightly bridge the gap between the learned subspace and label information, leading to a more discriminative feature representation. The superior classification performance using CoSpace is demonstrated on two different datasets, compared to using other state-of-art methods.

We performed transfer learning on homogeneous datasets in the considered MS-HS case in the sense that both data sources are optical images covering similar spectral ranges and thus the HS information can be transferred into the MS one linearly. The CoSpace’s ability in handling heterogeneous data sources remains limited due to its linearized modeling. In the future work, we will develop a more general system by integrating some powerful and emerging nonlinear tools (e.g., deep learning) into our framework.

In addition, we just assumed to share the same land-cover classes across MS and HS images in this paper. In reality, the number of land-cover classes in the large-scale MS scene might be usually more than the one in the overlapped area of MS and HS images. This naturally motivates us to generalize our model in the future work.

Appendix A Solution to problem (8) with respect to $\Theta$

The solution to problem (8) can be transferred to equivalently solve the problem (10) with ADMM. Considering the fact that the object function in Eq. (10) is not convex with respect to all variables simultaneously, but it is a convex problem regarding the separate variable when other variables are fixed, therefore we successively minimize $\mathscr{L}_{C}$ with respect to $\mathbf{\Theta},\mathbf{J},\mathbf{G},\mathbf{\Lambda}_{1},\mathbf{\Lambda}_{2}$ as follows:

Optimization with respect to $\mathbf{\Theta}$: The optimization problem for $\mathbf{\Theta}$ can be written as

[TABLE]

which has a closed-form solution:

[TABLE]

Optimization with respect to $\mathbf{J}$: The variable $\mathbf{J}$ can be estimated by solving the following problem:

[TABLE]

its analytical solution is given by

[TABLE]

Optimization with respect to $\mathbf{G}$: For $\mathbf{G}$, the optimization problem with orthogonal constraint can be formulated as

[TABLE]

which can be effectively solved using the strategy of splitting orthogonality constraints (SOC) [34] in two steps:

the first step is to perform the SVD factorization:

[TABLE]

the second step is to update $\mathbf{G}$ with satisfying orthogonal constraint:

[TABLE]

Lagrange multipliers ($\mathbf{\Lambda}_{1}$, $\mathbf{\Lambda}_{2}$) and penalty parameter ($\mu$) update: Before stepping into the next iteration, Lagrange multipliers need to be updated by

[TABLE]

and penalty parameter be updated by

[TABLE]

Acknowledgments

The authors would like to thank the Hyperspectral Image Analysis group and the NSF Funded Center for Airborne Laser Mapping (NCALM) at the University of Houston for providing the CASI University of Houston dataset. The authors would like to express their appreciation to Prof. D. Cai and Dr. C. Wang for providing MATLAB codes for LPP and manifold alignment algorithms.