Relation Network for Multi-label Aerial Image Classification

TL;DR

This paper introduces an attention-aware label relational reasoning network for multi-label aerial image classification, effectively modeling label dependencies and localizing discriminative regions to improve prediction accuracy.

Contribution

The proposed network combines label-wise feature extraction, attentional region localization, and relational reasoning, offering a novel, interpretable approach for multi-label aerial image classification.

Findings

Effective in extracting discriminative label-specific features

Successfully models label dependencies for improved accuracy

Validated on UCM and new AID multi-label datasets

Abstract

Multi-label classification plays a momentous role in perceiving intricate contents of an aerial image and triggers several related studies over the last years. However, most of them deploy few efforts in exploiting label relations, while such dependencies are crucial for making accurate predictions. Although an LSTM layer can be introduced to modeling such label dependencies in a chain propagation manner, the efficiency might be questioned when certain labels are improperly inferred. To address this, we propose a novel aerial image multi-label classification network, attention-aware label relational reasoning network. Particularly, our network consists of three elemental modules: 1) a label-wise feature parcel learning module, 2) an attentional region extraction module, and 3) a label relational inference module. To be more specific, the label-wise feature parcel learning module is designed for extracting high-level label-specific features. The attentional region extraction module aims at localizing discriminative regions in these features and yielding attentional label-specific features. The label relational inference module finally predicts label existences using label relations reasoned from outputs of the previous module. The proposed network is characterized by its capacities of extracting discriminative label-wise features in a proposal-free way and reasoning about label relations naturally and interpretably. In our experiments, we evaluate the proposed model on the UCM multi-label dataset and a newly produced dataset, AID multi-label dataset. Quantitative and qualitative results on these two datasets demonstrate the effectiveness of our model. To facilitate progress in the multi-label aerial image classification, the AID multi-label dataset will be made publicly available.

Tables

Table 1. TABLE I: The number of images for different object categories in the UCM multi-label dataset.

Category No.	Category Name	Training	Test	Total
1	bare soil	577	141	718
2	airplane	80	20	100
3	building	555	136	691
4	car	722	164	886
5	chaparral	82	33	115
6	court	84	21	105
7	dock	80	20	100
8	field	79	25	104
9	grass	804	171	975
10	mobile home	82	20	102
11	pavement	1047	253	1300
12	sand	218	76	294
13	sea	80	20	100
14	ship	80	22	102
15	tank	80	20	100
16	tree	801	208	1009
17	water	161	42	203
-	All	1680	420	2100

Table 2. TABLE II: The number of images for different object categories in the AID multi-label dataset.

Category No.	Category Name	Training	Test	Total
1	bare soil	1171	304	1475
2	airplane	79	20	99
3	building	1744	417	2161
4	car	1617	409	2026
5	chaparral	75	37	112
6	court	269	75	344
7	dock	221	50	271
8	field	175	39	214
9	grass	1829	466	2295
10	mobile home	1	1	2
11	pavement	1870	458	2328
12	sand	207	52	259
13	sea	177	44	221
14	ship	237	47	284
15	tank	87	21	108
16	tree	1923	483	2406
17	water	674	178	852
-	All	2400	600	3000

Table 3. TABLE III: Comparisons of the classification performance on UCM Multi-label Dataset (%).

Network	mean $F_1$	mean $F_2$	mean $p_e$	mean $r_e$	mean $p_l$	mean $r_l$
VGGNet [64]	78.54	80.17	79.06	82.30	86.02	80.21
VGG-RBFNN [31]	78.80	81.14	78.18	83.91	81.90	82.63
CA-VGG-BiLSTM [34]	79.78	81.69	79.33	83.99	85.28	76.52
AL-RN-VGGNet	85.70	85.81	87.62	86.41	91.04	81.71
GoogLeNet [59]	80.68	82.32	80.51	84.27	87.51	80.85
GoogLeNet-RBFNN [31]	81.54	84.05	79.95	86.75	86.19	84.92
CA-GoogLeNet-BiLSTM [34]	81.82	84.41	79.91	87.06	86.29	84.38
AL-RN-GoogLeNet	85.24	85.33	87.18	85.86	91.03	81.64
ResNet-50 [60]	79.68	80.58	80.86	81.95	88.78	78.98
ResNet-RBFNN [31]	80.58	82.47	79.92	84.59	86.21	83.72
CA-ResNet-BiLSTM [34]	81.47	85.27	77.94	89.02	86.12	84.26
AL-RN-ResNet	86.76	86.67	88.81	87.07	92.33	85.95

Table 4. TABLE IV: Example Images and Predicted labels on the UCM and AID Multi-label Dataset.

Samples from the UCM Multi-label Dataset	(a)	(b)	(c)	(d)	(e)
Ground Truths	building, car, court, grass, tree, and pavement	building, bare soil, pavement, and grass	car, tree, building, grass, and bare soil	pavement, grass, tree, and bare soil	car, pavement, and building
Predictions	building, car, court, grass, tree, and pavement	building, bare soil, pavement, and grass	tree, car, building, grass, bare soil, and pavement	pavement, grass, tree, and bare soil	car, pavement, and building
Samples from the AID Multi-label Dataset	(f)	(g)	(h)	(i)	(j)
Ground Truths	building, car, grass, tree, and pavement	car, bare soil, court, building, grass, tree, pavement, and water	building, car, tree, dock, grass, pavement, sea, and ship	bare soil, building, car, pavement, grass, tree, and water	court, building, car, bare soil, grass, tree, and pavement
Predictions	building, car, grass, tree, and pavement	car, bare soil, court, building, grass, tree, pavement, and water	building, car, tree, dock, grass, pavement, sea, water, and ship	bare soil, car, building, pavement, water, sand, tree, and grass	court, building, car, bare soil, grass, tree, and pavement

Table 5. TABLE V: Comparisons of the classification performance on AID Multi-label Dataset (%).

Network	mean $F_1$	mean $F_2$	mean $p_e$	mean $r_e$	mean $p_l$	mean $r_l$
VGGNet [64]	85.52	85.60	87.41	86.32	70.60	58.89
VGG-RBFNN [31]	84.58	85.99	84.56	87.85	62.90	69.15
CA-VGG-BiLSTM [34]	86.68	86.88	88.68	87.83	72.04	60.00
proposed AL-RN-VGGNet	88.09	88.31	89.96	89.27	76.94	68.31
GoogLeNet [59]	86.27	85.77	89.49	86.00	74.18	53.69
GoogLeNet-RBFNN [31]	84.85	86.80	84.68	89.14	65.41	72.26
CA-GoogLeNet-BiLSTM [34]	85.36	85.21	88.05	85.79	68.80	59.36
proposed AL-RN-GoogLeNet	88.17	88.25	90.03	88.77	77.92	69.50
ResNet-50 [60]	86.23	85.57	89.31	85.65	72.39	52.82
ResNet-RBFNN [31]	83.77	85.87	82.84	88.32	60.85	70.45
CA-ResNet-BiLSTM [34]	87.63	88.03	89.03	88.99	79.50	65.60
proposed AL-RN-ResNet	88.72	88.54	91.00	88.95	80.81	71.12

Table 6. TABLE VI: Comparison between different g θ l ​ m subscript 𝑔 subscript 𝜃 𝑙 𝑚 g_{\theta_{lm}} (%).

Dataset	$g_{{\theta }_{lm}}$	${\text{V*}}_{F_1}$	${\text{G*}}_{F_1}$	${\text{R*}}_{F_1}$	${\text{V*}}_{F_2}$	${\text{G*}}_{F_2}$	${\text{R*}}_{F_2}$
UCM mul.	MLP	82.11	83.02	85.36	81.99	84.02	86.09
	Conv.	85.70	85.24	86.76	85.81	85.33	86.67
AID mul.	MLP	87.79	84.92	87.10	87.74	86.97	86.83
	Conv.	88.09	88.17	88.72	88.31	88.25	88.54

Equations

$$\left[\begin{array}{c}{x}_l^{attn}\\ y_l^{attn}\end{array}\right]={\mathbf{M}}_{T_l}\left[\begin{array}{c}{x}_l\\ y_l\\ 1\end{array}\right],$$

$$\left[\begin{array}{c}{x}_l^{attn}\\ y_l^{attn}\end{array}\right]=\left[\begin{array}{ccc}{s}_{x_l}& 0& t_{x_l}\\ 0& s_{y_l}& t_{y_l}\end{array}\right]\left[\begin{array}{c}{x}_l\\ y_l\\ 1\end{array}\right],$$

$$\mathrm{LR}({\mathbf{A}}_l,{\mathbf{A}}_m)=f_{\varphi }(g_{{\theta }_{lm}}({\mathbf{A}}_l,{\mathbf{A}}_m)),$$

$$\mathrm{LR}({\mathbf{A}}_l,\ast )=f_{\varphi }(\sum _{m\ne l}{g}_{{\theta }_{lm}}({\mathbf{A}}_l,{\mathbf{A}}_m)),$$

$${\mathbf{M}}_{T_l}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\end{array}\right].$$

$$F_{\beta }=(1+{\beta }^2)\frac{p_e{r}_e}{{\beta }^2{p}_e+r_e},\beta =1,2,$$

$$p_e=\frac{T{P}_e}{T{P}_e+F{P}_e},r_e=\frac{T{P}_e}{T{P}_e+F{N}_e},$$

$$p_l=\frac{T{P}_l}{T{P}_l+F{P}_l},r_l=\frac{T{P}_l}{T{P}_l+F{N}_l},$$

$$\left[\begin{array}{cc}{x}_{BL}^{attn}& x_{TR}^{attn}\\ y_{BL}^{attn}& y_{TR}^{attn}\end{array}\right]={\mathbf{M}}_{T_l}\left[\begin{array}{cc}-1& 1\\ -1& 1\\ 1& 1\end{array}\right].$$

Full text

I Introduction

Recent advancements of remote sensing techniques have boosted the volume of attainable high-resolution aerial images, and massive amounts of applications, such as urban cartography [1, 2, 3, 4], traffic monitoring [5, 6, 7], terrain surface analysis [8, 9, 10, 11], and ecological scrutiny [12, 13], have benefited from these developments. For this reason, the aerial image classification has become one of the fundamental visual tasks in the remote sensing community and drawn a plethora of research interests [14, 15, 16, 17, 18, 19, 20, 21]. The classification of aerial images refers to assigning these images with specific labels according to their semantic contents, and a common hypothesis shared by many relevant studies is that an image should be labeled with only one semantic category, such as scene categories (see Fig. 1). Although such image-level labels [22, 23] are capable of delineating images from a macroscopic perspective, it is infeasible for them to provide a comprehensive view of objects in aerial images. To tackle this, huge quantities of algorithms have been proposed to identify each pixel in an image [24, 25, 26] or localize objects with bounding boxes [27, 28, 29]. However, the acquisition of requisite ground truths (i.e., pixel-wise annotations and bounding boxes) demands enormous expertise and human labors, which makes relevant datasets expensive and difficult to access. With this intention, multi-label image classification now attracts increasing attention in the remote sensing community [30, 31, 32, 33, 34] owing to that 1) a comprehensive picture of aerial image contents can be drawn, and 2) datasets required in this task are not expensive (only image-level labels are needed).

Fig. 1 illustrates the difference between image-level scene labels and object labels. As shown in this figure, although these four images are assigned with the same scene label, their multiple object labels vary a lot. It is worth noting that the identification of some objects can actually offer important cues to understand a scene more deeply. For example, the existence of building and pavement indicates a high probability that rivers in Fig. 1\alphalph and 1\alphalph are very close to areas with frequent human activities, while rivers in Fig. 1\alphalph and 1\alphalph are more likely in the wild due to the absence of human activity cues. In contrast, simply recognizing scene labels can hardly provide such information. Therefore, in this paper, we dedicate our efforts to explore an effective model for the multi-label classification of aerial images.

I-A Challenges of Identifying Multiple labels

In identifying multiple labels of an aerial image, two main challenges need to be faced with. One is how to extract semantic feature representations from raw images. This is crucial but difficult especially for high-resolution aerial images, as they always contain complicated spatial contextual information. Conventional approaches mainly resort to manually crafted features and semantic models [22, 35, 36, 37, 38], while these methods cannot effectively extract high-level semantics and lead to a limited performance in classification[23]. Hence an efficient high-level feature extractor is desirable.

The other challenge is how to take full advantage of label correlations to infer multiple object labels of an aerial image. In contrast to single-label classification, which mainly focuses on modeling image-label relevance, exploring and modeling label-label correlations plays a supplementary yet essential role in identifying multiple objects in aerial images. For instance, the presence of ships confidently infers the co-occurrence of water or sea, while the existence of a car suggests a high probability of the appearance of pavements. Unfortunately, such label correlations are scarcely addressed in the literature. One solution is to use a recurrent neural network (RNN) to learn label dependencies. However, this is done with a chain propagation fashion, and its performance heavily depends on the learning effectiveness of its long-term memorization. Moreover, in this way, label relations are modeled implicitly, which leads to a lack of interpretability.

Overall, an efficient multi-label classification model is supposed to be capable of not only learning high-level feature representations but also modeling label correlations effectively.

I-B Related Work

Zegeye and Demir [39] propose a multi-label active learning framework using a multi-label support vector machine (SVM), relying on both the multi-label uncertainty and diversity. Koda et al. [32] introduce a spatial and structure SVM for multi-label classification by considering spatial relations between a given patch and its neighbors. Similarly, Zeggada et al. [33] employ a conditional random field (CRF) framework to model spatial contextual information among adjacent patches for improving the performance of classifying multiple object labels.

With the development of computational resources and deep learning, very recent approaches mainly resort to deep networks for multi-label classification. In [31], the authors make use of a standard CNN architecture to extract feature representations and then feed them into a multi-label classification layer, which is composed of customized thresholding operations, for predicting multiple labels. In [40], the authors demonstrate that training a CNN for multi-label classification with a limited amount of labeled data usually leads to an underwhelming-performance model and propose a dynamic data augmentation method for enlarging training sets. More recently, Sumbul and Demir [41] propose a CNN-RNN method for identifying labels in multi-spectral images, where a bidirectional LSTM is employed to model spatial relationships among image patches. In order to explore inherent correlations among object labels, [34] proposes a CNN-LSTM hybrid network architecture to learn label dependencies for classifying object labels of aerial images. Besides, we also notice that several zero short learning researches focus on employing prior knowledge to model label relations. For instance, Sumbul et al. [42] apply an unsupervised word embedding model to encoding labels into word vectors, which are supposed to contain label semantics, and then model label relationships with these vectors. Lee et al. [43] propose to learn label relations from structured knowledge graphs observed from the real world.

I-C The Motivation of Our Work

In order to explicitly model label relations, we propose a label relational inference network for multi-label aerial image classification. This work is inspired by recent successes of relation networks in visual question answering [44], object detection [45], video classification [46], activity recognition in videos [47], and semantic segmentation [48]. A relation network is characterized by its inherent capability of inferring relations between an individual entity (e.g., a region in an image or a frame in a video) and all other entities (e.g., all regions in the image or all frames in the video). Besides, to increase the effectiveness of relational reasoning, we make use of a spatial transformer, which is often used to enhance the transformation invariance of deep neural networks [49], to reduce the impact of irrelevant semantic features.

More specifically, in this work, an innovative end-to-end multi-label aerial image classification network, termed as attention-aware label relational reasoning network, is proposed and characterized by its capabilities of localizing label-specific discriminative regions and explicitly modeling semantic label dependencies for the task. This paper’s contributions are threefold.

• •

We propose a novel multi-label aerial image classification network, attention-aware label relational reasoning network, which consists of three imperative components: a label-wise feature parcel learning module, an attentional region extraction module, and a label relational inference module. To our best knowledge, it is the first time that the idea of relation networks is employed to predict multiple object labels of aerial images, and experimental results demonstrate its effectiveness.

• •

We extract attentional regions from the label-wise feature parcels in a proposal-free fashion. Particularly, a learnable spatial transformer is employed to localize attentional regions, which are assumed to contain discriminative information, and then re-coordinate them into a given size. By doing so, attentional feature parcels can be yielded.

• •

To facilitate progress in the multi-label aerial image classification, we produce a new dataset, AID multi-label dataset, by relabeling images in the AID dataset [23]. In comparison with the UCM multi-label dataset [50], the proposed dataset is more challenging due to diverse spatial resolutions of images, more scenes, and more samples.

The remaining sections of this paper are organized as follows. Section II delineates three elemental modules of our proposed network, and Section III introduces experiments, where experimental setups are given and results are analyzed and discussed. Eventually, Section IV draws a conclusion of this paper.

II Methodology

II-A Network Architecture

As illustrated in Fig. 2, the proposed network comprises three components: a label-wise feature parcel learning module, an attentional region extraction module, and a label relational inference module. Let $L$ be the number of object labels and $l$ be the $l$-th label. The label-wise feature parcel learning module is designed to extract high-level feature maps $\bm{X}_{l}$ with $K$ channels, termed as feature parcel (for more details refer to Section II-B), for each label $l$. The attentional region extraction module is used to localize discriminative regions in each $\bm{X}_{l}$ and generate an attentional feature parcel $\bm{A}_{l}$, which is supposed to contain the most relevant semantics with respect to the label $l$. Finally, relations among $\bm{A}_{l}$ and all other label-wise attentional feature parcels are reasoned about by the label relational inference module for predicting the presence of the object $l$.

Details of the proposed network are introduced in the remaining sections.

II-B Label-wise Feature Parcel Learning

The extraction of high-level features is crucial for visual recognition tasks, and many recent studies adopt CNNs owing to their remarkable performance in learning such features [15, 51, 52, 53, 54, 55, 56]. Hence, we take a standard CNN as the backbone of the label-wise feature parcel learning module in our model. As shown in Fig. 2, an aerial image is first fed into a CNN (e.g., VGG-16), which consists of only convolutional and max-pooling layers, for generating high-level feature maps. Subsequently, these features are encoded into $L$ feature parcels for each label $l$ via a label-wise multi-modality feature learning layer. To implement this layer, we first employ a convolutional layer with $KL$ filters, whose size is $1\times 1$, to extract $KL$ feature maps. Afterwards, we divide these features into $L$ feature parcels, and each includes $K$ feature maps. That is to say, for each label, $K$ specific feature maps are learned, so-called feature parcel, to extract discriminative semantics after the end-to-end training of the whole network. We denote the feature parcel for label $l$ as $\bm{X}_{l}$ in the following statements.

In our experiment, we notice that $\bm{X}_{l}$ with a higher resolution is beneficial for the subsequent module to localize discriminative regions, as more spatial contextual cues are included. Accordingly, we discard the last max-pooling layer in VGG-16, leading to a spatial size of $14\times 14$ for outputs. Weights are initialized with pre-trained VGG-16 on ImageNet but updated during the training phase.

II-C Attentional Region Extraction Module

Although label-wise feature parcels can be directly applied to exploring label dependencies [34], less informative regions (see blue areas in Fig. 3) may bring noise and further reduce the effectiveness of these feature parcels. As shown in the left image of Fig. 3, weakly activated regions indicate a loose relevance to the corresponding label, while highlighted regions suggest a strong region-label relevance. To diminish the influence of unrelated regions, we employ an attentional region extraction module to automatically extract discriminative regions from label-wise feature parcels.

We localize and re-coordinate attentional regions from $\bm{X}_{l}$ with a learnable spatial transformer. Particularly, we sample a feature parcel $\bm{X}_{l}$ into a regular spatial grid $G_{\bm{X}_{l}}$ (cf. green dots in the left image of Fig. 3) according to the spatial resolution of $\bm{X}_{l}$ and regard pixels in $\bm{X}_{l}$ as points on the grid $G_{\bm{X}_{l}}$ with coordinates $(x_{l},y_{l})$. Similarly, we can define coordinates of a new grid, attentional region grid $G_{\bm{X}_{l}^{attn}}$ (see white dots in the middle image of Fig. 3), as $(x_{l}^{attn},y_{l}^{attn})$, and the number of grid points along with the height and width is equivalent to that of $G_{\bm{X}_{l}}$. As demonstrated in [49] that $G_{\bm{X}_{l}^{attn}}$ can be learned by performing spatial transformation on $G_{\bm{X}_{l}}$, $(x_{l}^{attn},y_{l}^{attn})$ can be calculated with the following equation:

[TABLE]

where $\bm{M}_{T_{l}}$ is a learnable transformation matrix, and grid coordinates, $x_{l}$ and $y_{l}$, are normalized to $[-1,1]$. Considering that this module is designed for localization, we only adopt scaling and translation in our case. Hence Eq. 1 can be rewritten as

[TABLE]

where $s_{x_{l}}$ and $s_{y_{l}}$ indicate scaling factors along x- and y-axis, respectively, and $t_{x_{l}}$ and $t_{y_{l}}$ represent how feature maps should be translated along both axes. Notably, since different objects distribute variously in aerial images, $\bm{M}_{T_{l}}$ is learned for each object label $l$ individually. In other words, extracted attentional regions are label-specific and capable of improving the effectiveness of label-wise features.

As to the implementation of this module, we first vectorize $\bm{X}_{l}$ with a flatten function and then employ a localization layer (e.g., a fully connected layer) to estimate elements in $\bm{M}_{T_{l}}$ from the vectorized $\bm{X}_{l}$. Afterwards, attentional region grid coordinates $(x_{l}^{attn},y_{l}^{attn})$ can be learned from $(x_{l},y_{l})$ with Eq. 2, and values of pixels at $(x_{l}^{attn},y_{l}^{attn})$ is able to be obtained from neighboring pixels by bilinear interpolation. Finally, the attentional region grid $G_{\bm{X}_{l}^{attn}}$ is re-coordinated to a regular spatial grid, which shares an identical structure with $G_{\bm{X}_{l}}$, for yielding the final attentional feature parcel $\bm{A}_{l}$.

II-D Label Relational Inference Module

Being the core of our model, the label relational inference module is designed to fully exploit label interrelations for inferring existences of all labels. Before diving into this module, we define the pairwise label relation as a composite function with the following equation:

[TABLE]

where the input is a pair of attentional feature parcels, $\bm{A}_{l}$ and $\bm{A}_{m}$, and $l$ and $m$ range from $1$ to $L$. The functions $g_{\theta_{lm}}$ and $f_{\phi}$ are used to reason about the pairwise relation between label $l$ and $m$. More specifically, the role of $g_{\theta_{lm}}$ is to reason about whether there exist relations between the two objects and how they are related. In previous works [44, 47], a multilayer perceptron (MLP) is commonly employed as $g_{\theta_{lm}}$ for its simplicity. However, spatial contextual semantics are not taken into account in this way. To address such issue, here, we make use of $1\times 1$ convolution instead of an MLP to explore spatial information. Furthermore, $f_{\phi}$ is applied to encode the output of $g_{\theta_{lm}}$ into the final pairwise label relation ${\rm LR}(\bm{A}_{l},\bm{A}_{m})$. In our case, $f_{\phi}$ consists of a global average pooling layer and an MLP, which finally yields the relation between label $l$ and $m$.

Following the motivation of our work, we infer each label by accumulating all related pairwise label relations, and the accumulated label relation for object label $l$ is defined as:

[TABLE]

where $*$ represents all attentional feature parcels except $\bm{A}_{l}$. Based on this formula, we implement the label relational inference module with the following steps (taking the prediction of label $l$ as an example): 1) $\bm{A}_{l}$ and every other attentional feature parcel are concatenated and fed into a $1\times 1$ convolutional layer, respectively. 2) Afterwards, a global average pooling layer is employed to transform $g_{\theta_{lm}}(\bm{A}_{l},\bm{A}_{m})$ into vectors, which are then element-wise added. 3) Finally, the output is fed into an MLP layer with trainable parameters $\phi$ to produce the accumulated label relation ${\rm LR}(\bm{A}_{l},*)$. Note that $g_{\theta_{lm}}$ is a learnable unit, which models pairwise relations using convolutions. Through the end-to-end training, it could be expected to learn data-driven label relations. Experiments in Section III-D and Section III-E have verified that learned label relations are in line with prior knowledge. Since we expect the model to predict probabilities, an activation function $\sigma$ is utilized to restrict each output digit to $[0,1]$. For label $l$, a digit approaching $1$ implies a high probability of its presence, while one closing [math] suggests the absence. Fig. 4 presents an visual illustration of the label relational inference module.

Compared to other multi-label classification methods, our model has three benefits:

The module can inherently reason about label relations as indicated by Eq. 3 and requires no particular prior knowledge about relations among all objects. That is to say, our network does not need to learn how to compute label relations and which object relations should be considered. All relations are automatically learned through a data-driven way and proven to meet the reality in our experiments. 2. 2.

The learning effectiveness is independent of long short-term memory, leading to increased robustness. This is because, in Eq. 4, accumulated label relations are calculated with a summation function instead of a chain architecture, e.g., an LSTM. 3. 3.

The function $g_{\theta_{lm}}$ is learned for each object label pair $l$ and $m$ separately, which suggests that pairwise label relations are encoded in a specific way. Besides, our implementation of $g_{\theta_{lm}}$ can extend the applicability of relational reasoning compared to using an MLP.

Since [34] shares the same design philosophy that modeling label relations is crucial, here we emphasize two differences between our network and [34]: 1) the proposed network learns to extract discriminative regions as label-wise features for modeling label relations (cf. Section II-C) instead of directly using entire feature maps as in [34]; 2) the proposed label relation inference module encodes label relations explicitly with composite functions, while in [34], label relations are modeled implicitly via an RNN whose effectiveness depends heavily on the learning effect of long-term memorization. Quantitative comparisons between these two approaches are shown in the following section.

III Experiments and Discussion

In this section, we conduct experiments on the UCM [50] and proposed AID multi-label dataset for evaluating our model. Specifically, Section III-A presents a description of these two datasets. Afterwards, we introduce training strategies and thoroughly discuss experimental results in the subsequent subsections.

III-A Dataset Introduction

III-A1 UCM multi-label dataset

UCM multi-label dataset [50] is reproduced by assigning all aerial images collected in UCM dataset [22] with newly defined object labels. The number of all candidate object labels is 17: building, sand, dock, court, tree, sea, bare soil, mobile home, ship, field, tank, water, grass, pavement, chaparral, and car. It is worth noting that labels, such as tank, airplane, and building, exist in both [22] and [50] while at different levels. In [22], such terms are considered as scene-level labels due to the fact that related images can be characterized and depicted by them, while in [50], they mean objects that may present in aerial images.

As to properties of images in this dataset, the spatial resolution of each sample is one foot, and the size is $256\times 256$ pixels. All images are manually cropped from aerial imagery contributed by the National Map of the U.S. Geological Survey (USGS), and there are 2100 images in total. For each object category, the number of images is listed in Table I. Besides, 80% of image samples per scene class are selected to train our model, and the other 20% of images are used to test our model. Numbers of images assigned to training and test sets with respect to all object labels are available in Table I as well. Some visual examples are shown in Fig. 5.

III-A2 AID multi-label dataset

In order to further evaluate our network and meanwhile promote progress in the area of multi-class classification of high-resolution aerial images, we produce a new dataset, named AID multi-label dataset, based on the widely used AID scene classification dataset [23]. The AID dataset consists of 10000 high-resolution aerial images collected from worldwide Google Earth imagery, including scenes from China, the United States, England, France, Italy, Japan, and Germany. In contrast to the UCM dataset, spatial resolutions of images in the AID dataset vary from 0.5 m/pixel to 8 m/pixel, and the size of each aerial image is $600\times 600$ pixels. Besides, the number of images in each scene category ranges from 220 to 420. Overall, the AID dataset is more challenging compared to the UCM dataset.

Here, we manually relabel some images in the AID dataset. With extensive human visual inspections, 3000 aerial images from 30 scenes in the AID dataset are selected and assigned with multiple object labels, and the distribution of samples in each category is shown in Table II. Besides, 80% of all images are taken as training samples, while the rest is used for testing our model. Several example images are shown in Fig. 6.

III-B Training Details

As to the initialization of our network, different modules are done in different ways. For the label-wise feature parcel learning module, we initialize the backbone and weights in other convolutional layers with a pre-trained ImageNet [57] model and a Glorot uniform initializer, respectively. Regarding the attentional region extraction module, we initialize the transformation matrix in Eq. 1 as an identical transformation,

[TABLE]

In the label relational inference module, weights in both $f_{\phi}$ and $g_{\theta_{lm}}$ are initialized with a Glorot uniform initializer and updated during the training phase. Notably, the entire network is trained in an end-to-end manner, and weights in the backbone are fine-tuned as well.

In our case, multiple labels are encoded into multi-hot binary sequences instead of one-hot vectors widely used in single-label classification tasks. The length of such multi-hot binary sequence is identical to the number of total object categories, i.e., 17 in our case, and as to each digit, 0 suggests an absent object, while 1 indicates the presence of its corresponding object label. Accordingly, we define the network loss as the binary cross-entropy. Besides, Adam with Nesterov momentum [58], which shows faster convergence than stochastic gradient descent (SGD) for our task, are selected and its parameters are set as recommended [58]: $\epsilon=1e-08$, $\beta_{1}=0.9$, and $\beta_{2}=0.999$. The learning rate is initially defined as $1e-04$ and decayed by a factor of 10 if the validation loss fails to decrease. Notably, we randomly select 10% of the training samples as the validation set. That is, during the training procedure, we use 90% of the training samples to learn network parameters.

Our model is implemented on TensorFlow-1.12.0 and trained for 100 epochs. The computational resource is an NVIDIA Tesla P100 GPU with a 16GB memory. As a compromise between the training speed and GPU memory capacities, we set the size of training batches as 32. To avoid overfitting, the training progress is terminated once the validation loss increases continuously in five epochs.

III-C Experimental Setup

To fully explore the capacity of our proposed network, we extend our researches by replacing the backbone with GoogLeNet (Inceptionv3) [59] and ResNet (ResNet-50 in our case) [60]. Specifically, we adapt GoogLeNet by removing global average pooling and fully-connected layers as well as reducing the stride of convolutional and pooling layers in “mixed8” to 1 to improve the spatial resolution. Besides, in order to preserve receptive fields of subsequent convolutional layers, filters in “mixed9” are replaced with atrous convolutional filters, and the dilation rate is defined as 2. Regarding ResNet, we set the convolution stride and dilation rate of filters as 1 and 2, respectively, in the last residual block. Global average pooling and fully-connected layers are removed as well.

In our experiments, we compare the proposed attention-aware label relational reasoning network (AL-RN-CNN) with the following competitors: a standard CNN, CNN-RBFNN [31], and CA-CNN-BiLSTM [34]. Regarding the CNN, we replace its last softmax layer, designed for single-label classification, with a sigmoid layer to produce multi-hot sequences. For the CA-CNN-BiLSTM, we follow the experimental configurations in [34]. Specifically, we first initialize the feature extraction module of CA-CNN-BiLSTM and weights in the bidirectional LSTM layer with CNNs pre-trained on ImageNet dataset and random values from -0.1 to 0.1, respectively. Afterwards, we fine-tune the entire network in the training phase with Nestro Adam optimizer, and the initial learning rate is set as $1e-04$. The loss is calculated with the binary cross-entropy, and the size of training batches is 32. Notably, for all models, output sequences are binarized with a threshold of 0.5 to generate final predictions.

III-D Results on the UCM Multi-label Dataset

III-D1 Quantitative analysis

In our experiment, we employ $F_{1}$ [61] and $F_{2}$ [62] scores as evaluation metrics to quantitatively assess the performance of different models. Specifically, these two $F$ scores are calculated with the following equation:

[TABLE]

where $p_{e}$ indicates the example-based precision and recall [63] of predictions. Formulas of calculating $p_{e}$ and $r_{e}$ are:

[TABLE]

where $TP_{e}$ (example-based true positive) indicates the number of correctly predicted positive labels in an example, while $FP_{e}$ (example-based false positive) denotes the number of those failed to be recognized. Besides, $FN_{e}$ (example-based false negative) represents the number of incorrectly predicted negative labels in an example. Here, an example stands for an aerial image and its associated multiple labels.

To evaluate our network comprehensively, we take mean $F_{1}$ and $F_{2}$ score as principal indexes. Moreover, we also report mean $p_{e}$ and mean $r_{e}$. In addition to the example-based perspective, label-based precision and recall are also considered and calculated with:

[TABLE]

to demonstrate the performance of networks from the perspective of each object label.

Table III exhibits experimental results on the UCM multi-label dataset. We can observe that our model surpasses all competitors on the UCM multi-label dataset with variant backbones. Specifically, AL-RN-VGGNet increases mean $F_{1}$ and $F_{2}$ scores by 7.16% and 5.64%, respectively, in comparison with VGGNet. Compared to CA-VGG-BiLSTM, which resorts to employing a bidirectional LSTM structure for exploring label dependencies, our network obtains an improvement of 5.92% in the mean $F_{1}$ score. Besides, although CA-VGG-BiLSTM is superior to VGGNet in both mean $F_{1}$ and $F_{2}$ scores, it achieves decreased mean precisions and recalls. In contrast, AL-RN-VGGNet outperforms VGGNet not only in mean $F_{1}$ and $F_{2}$ scores but also in mean example- and label-based precisions and recalls. For another backbone, GoogLeNet, our network gains the best mean $F_{1}$ and $F_{2}$ scores. As shown in Table III, AL-RN-GoogLeNet increases the mean $F_{1}$ score by 4.56% and 3.42% with respect to GoogLeNet and CA-GoogLeNet-BiLSTM, respectively. For the mean $F_{2}$ score and precisions, our model also surpasses other competitors, which proves the effectiveness and robustness of our method. AL-RN-ResNet achieves the best mean $F_{1}$ score, 0.8676, and $F_{2}$ score, 0.8667, in comparison with all other models. Furthermore, it obtains the best mean example-based precision, 0.8881, and label-based precision, 0.9233, and recall, 0.8595. To summarize, comparisons between AL-RN-CNN and other models demonstrate the effectiveness of our network. Moreover, comparisons between AL-RN-CNN and CA-CNN-BiLSTM illustrate that the composite function-based proposed model performs better than a BiLSTM framework in terms of both accuracy and robustness. Reasons could be that: 1) a chain-like BiLSTM architecture might suffer from the error propagation [41] and thus is sensitive to the order of predictions, while in our network, all pair-wise label relations are encoded separately and the final summation function is order invariant [44]. 2) a BiLSTM-based structure models label relations implicitly, whereas our network encodes such relations in an explicit and direct way. Table IV presents several example predictions from the UCM multi-label dataset. As a supplementary study, we evaluate the robustness of our proposed model by performing cross-validation in the training phase. More specifically, we randomly divide training samples into five folds and train our best-performed model, i.e., AL-RN-ResNet, five times. For each training progress, we select one of five folds as the validation set and train our model with the remaining four folds. We observe that variances of mean $F_{1}$ and $F_{2}$ scores are 0.38% and 0.71%, respectively. Compared to improvements brought by our network, variances are limited, and this demonstrates the robustness of our proposed network.

III-D2 Qualitative analysis

In order to figure out what is going on inside our network, we further visualize features learned from each module and validate the effectiveness of the proposed network in a qualitative manner. In Fig. 7, a couple of feature parcels regarding bare soil, building, car, pavement, court, and tank is displayed for several example images. Note that for $K$ feature maps in each feature parcel, we select the most strongly activated one as the representative. We can observe that discriminative regions related to positive labels are highlighted in these feature maps, while less informative regions are weakly activated. As an exception, the feature map at the bottom left of Fig. 7 shows that the baseball field is misidentified as tanks, which may lead to incorrect predictions.

For evaluating the localization ability of the proposed network, we visualize attentional regions learned from the second module. Coordinates of bottom left (BL) and top right (TR) corners of attentional region grids are calculated with the following equation:

[TABLE]

Fig. 8 shows some examples of learned attentional regions. As we can see, most attentional regions concentrate on areas covering objects of interest. Besides, it is noteworthy that even objects are distributed dispersedly, the learned attentional regions can still cover most of them, e.g., buildings in Fig. 8\alphalph and cars in 8\alphalph.

Furthermore, learned pairwise label relations are visualized in the format of matrix, where an element at $(l,m)$ indicates ${\rm LR}(\bm{A}_{l},\bm{A}_{m})$. Fig. 9 exhibits some examples for the four scenes in Fig. 8. In these examples, we take only positive object labels into consideration and perform normalization alongside each row to yield a distinct visualization of “label relations”. Since $m$ differs from $l$, we assign null values to diagonal elements and mark them as white color in Fig. 9. It can be seen that in Fig. 9\alphalph and 9\alphalph, relations between car and pavement contribute significantly to predicting presences of both car and pavement. Besides, Fig. 9\alphalph shows that the existence of tree highly suggests the presence of bare soil, but not vice versa. These observations illustrate that even without prior knowledge, the proposed network can reason about relations, that are in line with the reality.

III-E Results on the AID Multi-label Dataset

III-E1 Quantitative analysis

To further evaluate the proposed network, we report experimental results on the AID multi-label dataset. Evaluation metrics here are the same as those in previous experiments, and results are presented in Table V. As we can observe, the proposed AL-RN-CNN behaves superior to all competitors in most of the metrics. To be more specific, AL-RN-VGGNet improves the mean $F_{1}$ and $F_{2}$ score by 2.57% and 2.71%, respectively, compared to the baseline model. In comparison with CA-VGG-BiLSTM, our network gains an improvement of 1.41% in the mean $F_{1}$ score and 1.43% in the mean $F_{2}$ score. Regarding the other two backbones, similar phenomena can be observed as well. AL-RN-GoogLeNet achieves the highest mean $F_{1}$ and $F_{2}$ score, 0.8817 and 0.8825, compared to GoogLeNet and CA-GoogLeNet-BiLSTM, while AL-RN-ResNet surpasses the second-best model by 1.09% and 0.51% in the mean $F_{1}$ and $F_{2}$ score, respectively. Besides, it is noteworthy that although CA-GoogLeNet-BiLSTM shows a decreased performance compared to the baseline model, our network still achieves higher scores in all metrics. Moreover, we notice that the proposed AL-RN-CNNs outperform baseline CNNs by a large margin in the mean label-based recall, and the maximum improvement can reach 18.30%. In conclusion, these comparisons suggest that explicitly modeling label relations can improve the robustness and retrieval ability of a network. Several example predictions on the AID multi-label dataset are presented in Table IV.

III-E2 Qualitative analysis

To dive deep into the model, we visualize label-specific features and attentional regions in Fig. 10 and 11, respectively. In Fig. 10, representative feature maps in various feature parcels for bare soil, building, car, pavement, tree, and water are displayed. As shown here, regions with label-related semantics are highlighted, while less informative regions present weak activations. For instance, regions of ponds are considered as discriminative regions for identifying water. Residential and industrial areas are strongly activated in feature maps for recognizing building. In Fig. 11, it can be observed that attentional regions learned from our network are able to capture areas of semantic objects, such as cars and trees. We also note that some attentional regions in Fig. 11 are coarser than those in Fig. 8, which is because the AID multi-label dataset has a lower spatial resolution.

Furthermore, pairwise relations among positive labels are visualized in Fig. 12. As shown in Fig. 12\alphalph, 12\alphalph, and 12\alphalph, existences of both tree and pavement contribute significantly to the identification of car, while the occurrence of car only suggests a high probability that pavement presents. Strong pairwise relations between building and other labels, e.g., car, pavement, and tree, indicate that the presence of building can heavily assist in predicting those labels.

III-F Discussion on the Relational Inference Module

Regarding the relational inference module, the function $g_{\theta_{lm}}$ is an important component, which reasons about relations between two objects. Hence, in this subsection, we discuss about different implementations of $g_{\theta_{lm}}$. Specifically, we compare our AL-RN-CNN with LR-CNN [65], which employs a global average pooling layer and an MLP as $g_{\theta_{lm}}$, on both the UCM and AID multi-label datasets. Experimental results are reported in Table VI. As shown in this table, our network gains the best mean $F_{1}$ and $F_{2}$ score on both datasets with variant backbones. AL-RN-VGGNet achieves the highest improvements of 3.59% and 3.82% for the mean $F_{1}$ and $F_{2}$ score, respectively, compared to LR-VGGNet on the UCM multi-label dataset. AL-RN-GoogLeNet increases the mean $F_{1}$ and $F_{2}$ score by 3.25% and 1.28%, respectively, in comparison with LR-ResNet on the AID multi-label dataset. Moreover, AL-RN-CNN can encode label relations through various fields of view by simply changing the size of convolutional filters in $g_{\theta_{lm}}$.

IV Conclusion

In this work, we propose a novel aerial image multi-label classification network, namely attention-aware label relational reasoning network. This network comprises three components: a label-wise feature parcel learning module, an attentional region extraction module, and a label relational inference module. To be more specific, the label-wise feature parcel learning module is designed to learn high-level feature parcels, which are proven to encompass label-relevant semantics, and the attentional region extraction module further generates finer attentional feature parcels by preserving only features located in discriminative regions. Afterwards, the label relational inference module reasons about pairwise relations among all labels and exploit these relations for the final prediction. In order to assess the performance of our network, experiments are conducted on the UCM multi-label dataset and a newly proposed AID multi-label dataset. In comparison with other deep learning methods, our network can offer better classification results. In addition, we visualize extracted feature parcels, attentional regions, and relation matrices for demonstrating the effectiveness of each module in a qualitative way. Looking into the future, such network architecture has several potentials, e.g., weakly supervised object detection and semantic segmentation.