Person Re-identification in Aerial Imagery

TL;DR

This paper introduces a new large-scale dataset for person re-identification in aerial imagery captured by UAVs and proposes a novel feature representation method that improves re-identification performance in this challenging setting.

Contribution

The paper presents the PRAI-1581 dataset for aerial person ReID and a subspace pooling method for discriminative feature learning, advancing research in UAV-based surveillance.

Findings

ReID in aerial imagery is highly challenging.

Proposed method outperforms existing approaches.

Dataset facilitates future UAV surveillance research.

Abstract

Nowadays, with the rapid development of consumer Unmanned Aerial Vehicles (UAVs), visual surveillance by utilizing the UAV platform has been very attractive. Most of the research works for UAV captured visual data are mainly focused on the tasks of object detection and tracking. However, limited attention has been paid to the task of person Re-identification (ReID) which has been widely studied in ordinary surveillance cameras with fixed emplacements. In this paper, to facilitate the research of person ReID in aerial imagery, we collect a large scale airborne person ReID dataset named as Person ReID for Aerial Imagery (PRAI-1581), which consists of 39,461 images of 1581 person identities. The images of the dataset are shot by two DJI consumer UAVs flying at an altitude ranging from 20 to 60 meters above the ground, which covers most of the real UAV surveillance scenarios. In addition, we propose to utilize subspace pooling of convolution feature maps to represent the input person images. Our method can learn a discriminative and compact feature representation for ReID in aerial imagery and can be trained in an end-to-end fashion efficiently. We conduct extensive experiments on the proposed dataset and the experimental results demonstrate that re-identify persons in aerial imagery is a challenging problem, where our method performs favorably against state of the arts. Our dataset can be accessed via \url{https://github.com/stormyoung/PRAI-1581}.

Tables

Table 1. TABLE I: Detailed settings comparison between our PRAI-1581 and other related datasets.

	Market1501 [5]	MRP(Dataset1) [29]	MRP(Dataset2) [29]	AVI [26]	PRAI-1581
#Images	32668	1501	4096	10863	39461
#ID	1501	23	28	5124	1581
Platform	fixed emplacement	UAV	UAV	UAV	UAV
Height				2$\sim$8m	20$\sim$60m
Camera	fixed	mobile	mobile	mobile	mobile
#UAVs	0	3	6	$\times$	2

Table 2. TABLE II: Results of Baseline and State-Of-The-Art ReID Methods on PRAI-1581 Dataset. ID denotes Identification Loss, TL denotes batch hard Triplet Loss, STL denotes Strong Triplet Loss.

method	mAP	rank-1
ID	31.47	42.62
TL [6]	36.49	47.47
PCB [46]	37.15	47.47
STL [46]	37.13	47.49
SVDNet [22]	36.70	46.10
AlignedReID [47]	37.64	48.54
PCB+RPP [46]	38.45	48.07
MBC [48]	22.83	30.05
2Stream [49]	37.02	47.79
Part-align [50]	32.86	43.14
DSR [51]	39.14	51.09
IDE [52]	32.90	43.90
DCGAN [28]	28.82	38.93
Deep Embedding [32]	14.73	21.36
MGN [53]	40.86	49.64
OSNET [54]	42.10	54.40

Table 3. TABLE III: Results of the proposed subspace pooling method on PRAI-1581 dataset. TL denotes Triplet Loss, ID denotes Identification Loss, SP denotes subspace pooling.

method	Single-query			Multi-query
	mAP	Rank-1	F-score	mAP	Rank-1	F-score
ID	31.47	42.46	32.38	39.69	51.95	40.34
ID+SP	37.88	48.33	38.34	43.22	54.95	43.43
gains	6.41	5.87	5.96	3.53	3.00	3.09
TL	36.49	47.47	38.49	43.31	54.95	43.43
TL+SP	39.58	49.79	39.82	44.75	56.49	44.73
gains	3.09	2.32	1.33	1.44	1.54	1.30

Table 4. TABLE IV: The results of different partition ratios of training set over test set.

ratio	Single-query		Multi-query
	mAP	Rank-1	mAP	Rank-1
2:8	18.88	26.49	23.53	32.95
3:7	25.03	33.44	31.17	42.14
4:6	31.44	41.93	37.84	49.59
5:5	39.58	49.79	45.23	54.74
6:4	44.30	54.20	51.58	62.57
7:3	50.33	60.40	58.89	69.63
8:2	59.36	69.96	66.94	77.57

Table 5. TABLE V: The results of different backbone networks.

network	Single-query		Multi-query
	mAP	Rank-1	mAP	Rank-1
GoogLeNet	34.86	44.94	38.80	51.32
ResNet-34	35.99	47.21	41.03	52.86
ResNet-50	39.58	49.79	44.75	56.49
ResNext-50	37.30	48.13	43.00	54.39
DenseNet-161	43.05	54.76	50.97	62.41
MobileNet	33.17	44.55	38.95	51.12
ShuffleNet	19.02	26.71	35.35	47.70

Table 6. TABLE VI: Results on Market-1501, DukeMTMC and CUHK-03 datasets.

Dataset	Baseline			Baseline+SP			gains
	mAP	Rank-1	F-score	mAP	Rank-1	F-score	mAP	Rank-1	F-score
Market-1501	71.44	86.49	67.36	72.51	86.52	68.61	1.07	0.03	1.25
DukeMTMC	60.83	77.51	58.7	63.88	79.35	61.39	3.05	1.84	3.39
CUHK03	49.36	53.93	49.02	51.11	55.86	51.01	1.75	1.93	1.99

Table 7. TABLE VII: The results of different output sizes about 1 × \times 1 convolution layer.

size	mAP	Rank-1	F-score
64	33.51	43.72	34.39
128	36.41	46.35	36.92
256	37.62	48.24	38.05
512	39.58	49.79	39.82

Equations

$$\mathbf{A}=\mathbf{US}{\mathbf{V}}^T=\sum _{i=1}^c{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$$

$$f(\mathbf{A})={\mathbf{U}}_k=[{\mathbf{u}}_1,{\mathbf{u}}_2,\cdots ,{\mathbf{u}}_k]$$

$${\mathcal{L}}_{CroEn}=-\frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{y}_{i,t}\log p_{i,t}$$

$$\displaystyle\mathcal{L}_{Tri}=\sum_{i=1}^{P}\sum_{a=1}^{K}\max\big{(}0,{D_{a,p}^{*}}-{D_{a,n}^{*}}+m\big{)}$$

Full text

I Introduction

Person Re-identification (ReID) aims to re-identify the same individual from other non-overlapping cameras [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], which is a key technique to achieve the tasks of long-term person tracking, cross camera tracking, person matching etc. in the field of intelligent visual surveillance. Nowadays, with the rapid development of the consumer Unmanned Aerial Vehicles (UAVs), visual surveillance by utilizing the UAV platform has been booming and become a necessary supplementary tactic for the tradition surveillance situation with fixed camera emplacements.

Recently, more and more efforts have been devoted into the study of intelligent aerial surveillance [14, 15, 16, 17] from both the industrial and academia. However, most of the research works are mainly focused on the tasks of object detection [18, 19, 20] and tracking [21]. There is limited attention paid to the task of person ReID, which is probably because there does not exist a large scale publicly available person ReID dataset facing to the real UAV surveillance scenarios, although ReID is as important as detection and tracking in UAV-based visual surveillance. As well as we all know, the annotation of the ReID task is much labor-intensive as one needs to annotate not only the bounding box positions, but also the ID numbers of each person.

In this paper, we collect a large scale airborne person ReID dataset named as Person ReID in Aerial Imagery (PRAI-1581), which consists of 39,461 images of 1581 person identities. The images of the dataset are shot by two DJI consumer UAVs flying at an altitude ranging from 20 to 60 meters above the ground, which covers most of the real UAV surveillance scenarios. The two UAVs are controlled by two different pilots to fly in different areas with non-overlapping monitoring field. It is worth noting that, due to the variable altitude of the flight, the adjustable tilt angle of the camera, and the free rotation of the fuselage, persons have diverse resolutions, viewpoints and poses in a single UAV, and the entirety scenarios are even more complicated since we have two UAVs controlled independently. Compared to the person ReID task in ordinary surveillance scenario with fixed camera emplacement, it is much more challenging to match persons in aerial imagery. Figure 1 shows some example images of our collected dataset. The collection and annotation of our dataset takes us more than 800 hours with 10 well-trained annotators. Besides the person ReID task, our dataset can also be used as a benchmark dataset for person detection and tracking in aerial images. We will release our dataset and benchmark to the public to facilitate researches in this area.

On the other hand, learning discriminative and invariant features is the key to visual recognition, especially for the ReID task where intra-person visual differences are often larger than inter-person similarities. Currently, almost all state-of-the-art ReID methods are based on deep learning. Typically, such ReID methods try to embed person images into a discriminative and compact feature space, then compute the similarities between the query and gallery images. In those methods, features are represented by the last fully-connected (FC) layer. Usually, the last FC layer extracts very high abstract features, and there is no explicit orthognality constraint on the weight vectors within the FC layer. Thus the weight vectors are easily to suffer from being highly correlated, leading to non-compact and redundant features where neurons of FC layer are correlated with each other [22]. To reduce feature redundancies which may hamper the final ranking results in ReID evaluation, Sun et al. [22] propose SVDNet which uses the left unitary matrix and singular value matrix of the SVD results to replace the original weight vectors. Although the ReID performance can be consistently improved through the Restraint and Relaxation Iteration (RRI) training scheme, it might be not optimal as it breaks the end-to-end training mechanism and has to obtain the solution in an alternating variable way.

In this paper, we try to handle the problem mentioned above by utilize subspace pooling [23, 24] which transforms the CNN feature matrix into a compact orthogonal matrix consists of its principal singular vectors. Such orthogonal matrices lie on the Grassmann manifold [25], a Riemannian manifold where a point represents the subspace of the Euclidean space. The subspace pooling layer can be plugged into any backbone networks and trained via an end-to-end back-propagation learning method, leading to a compact and discriminative feature representations of person images shot by UAVs.

We make thorough analysis of our collected PRAI-1581 dataset, show its new features against previous datasets and conduct extensive experiments on it. Specifically, we test 12 state-of-the-art person ReID methods and the subspace pooling method on PRAI-1581. Experimental results show that our method performs favorably against others. Nevertheless, all the performances are far from satisfactory which demonstrate that re-identify persons in aerial imagery is very challenging and further efforts should be made in this research direction.

The contributions of this paper can be summarized from the following aspects:

Person ReID Dataset in Aerial Imagery. Public datasets such as Market1501 and CUHK03 are shot under the fixed surveillance cameras. Our dataset is shot by UAVs which are flying at an altitude ranging from 20 to 60 meters. The PRAI-1581 dataset contains over 39,461 images of 1581 person identities and to our best knowledge, it is the first large person ReID dataset in aerial imagery facing to the real UAV surveillance scenarios. 2. 2.

Subspace Pooling for Person ReID. We propose to utilize subspace pooling of CNN features to represent the input person images by a discriminative and compact feature representation, which achieves state-of-the-art performance on our collected dataset. The method can make the entries of the feature independent and the neurons learn de-correlated descriptors for person images. 3. 3.

Comprehensive Experimental Results. We conduct thorough experiments on our dataset, including typical baseline, representative state-of-the-art ReID methods, and our subspace pooling method. To test the practical performance of the ReID on our dataset.

The rest of the paper is organized as follows: Section II reviews the related works of our paper. Section III describes our PRAI dataset in detail. In Section IV we elaborate the approach of subspace pooling with SVD. Section V gives the experimental results. In SectionVI we draw the conclusion of the paper.

II Related works

In this section, we review the related works from the following three aspects: aerial datasets for vision tasks, person ReID methods and compact feature learning.

II-A Aerial Datasets for Vision Tasks

In recent years, more and more aerial datasets have emerged, featuring at scale diversity, perspective specificity, multi-directional and high background complexity. DOTA [14] is an object detection dataset in aerial images which are mainly collected from the Google Earth, satellite JL-1, and satellite GF-2 of the China Centre for Resources Satellite Data and Application. It contains 0.4 million annotated object instances within 16 categories. NWPU VHR [17] is an aerospace remote sensing dataset including 650 targets and 150 scene images, and the main purpose of the dataset is serving as an detection benchmark, too. UAV123 [16] is a video dataset taken by UAVs and aimed to serve as a target tracking dataset. VisDrone [15] is a visual object detection and tracking dataset, serving as a benchmark dataset for a challenge named “Vision Meets Drone: A Challenge” organized as a workshop in ECCV and ICCV. AVI [26] dataset is collected to deal with the task of violent individuals identification, which is composed of 2000 aerial images and annotated with 14 key-points for each person. As for the field of Person ReID, the datasets commonly used are Market1501 [5], MARS [27], DukeMTMC-reID [28] and CUHK03 [1] etc., which are shot by the cameras with fixed emplacement with fixed angle. MRP [29] is a public dataset for mobile re-identification. The purpose of the dataset is to verify the ReID algorithms on mobile platform. Although it is taken by UAVs, the UAVs are flying at a relatively low altitude (more or less higher than normal person height) and it does not match the practical UAV surveillance scenarios.

In this paper, we collect PRAI-1581 dataset for person ReID task facing to the real UAV surveillance scenarios. Our dataset provides images from both all sorts of shooting angles and diverse resolutions duo to the cameras and the UAV platform that can be controlled flexibly, which greatly meets the research needs of person ReID in aerial imagery. Meanwhile, it is the first aerial person ReID dataset facing to real surveillance scenarios to our best knowledge.

II-B Person ReID Methods

Person ReID has been attracting a lot of research efforts in the community. Traditionally, it has been addressed to deal with from two aspects: the feature representation and metric learning problem, while current state-of-the-art ReID methods are all based on deep learning techniques [30, 31, 32, 33, 34, 35, 36, 9], which jointly optimizes the two phases together. Zhou et al. [7] proposed a set to set distance to learn discriminative and stable feature representations and to effectively find out the matched target to the probe object among various candidates. Ge et al. [37] proposed to utilize pose information and adversarial learning to distill identity-related and pose-unrelated feature representation.

In addition, unsupervised and cross-modal ReID methods, also have achieved promising performance in recent years. Li et al. [38] proposed an unsupervised deep ReID approach, which can incrementally discovered and exploited the underlying discriminative information for ReID. Yu et al. [39] introduced a deep model for the multi-label learning for unsupervised ReID, which learns a discriminative embedding for the unlabeled target domain through the soft multi-label guided hard negative mining. Shen et al. [40] proposed SGGNN and utilized the relationship between probe-gallery pairs to provide more consistent information. Wei et al. [41] adopted GAN to bridge the domain gap between different person ReID dataset. Li et al. [42] utilized a new spatio-temporal attention module based on person video to allow useful information to be extracted. In this work, we study the effect of subspace pooling on person ReID task, which takes full advantage of the matrix orthogonality to obtain discriminative and invariant features through an end-to-end learning way.

II-C Compact Feature Learning

Lots of research works have been focusing on learning compact and non-redundant feature descriptors which have many good merits [43, 23, 44, 22, 45, 24]. Chan et al. [45] proposed a very simple deep learning framework called PCANet for image classification, which learned orthogonal projection to produce the filters. Xie et al. [44] utilized the regularization effect of orthogonality to improve the classification accuracy. Sun et al. [43] quantitatively analyzed the influence of features on person ReID accuracy and found that the associations between different features also impacted the results. Wei et al. [23, 24] proposed subspace pooling layer to learn discriminative and invariant features. Sun et al. [22] exploited the SVDNet to generate de-correlated descriptors suitable for person retrieval, which directly benefit the feature learning process. Although the ReID performance can be consistently improved by SVDNet, it might be not optimal as it breaks the end-to-end training procedure and has to obtain the solution in the alternating variable way. In this work, different from SVDNet, we incorporate subspace pooling layer to transform convolution feature maps into compact feature descriptors through an end-to-end learning method.

III DATASET

In this section, we elaborate the features of our PRAI-1581 dataset, including the collection and annotation process in detail.

III-A Dataset Collection

We use two UAVs to shoot videos in two adjacent areas where there are non-overlap visible areas. PRAI-1581 contains pairing data from two flights which are in unconstrained and heavily crowded outdoor environments. In order to acquire more sufficient video including diverse viewpoints and backgrounds, we adopted the hovering, cruising, and rotating sport models in the process of controlling UAV. We collect about 60 pairs of videos and then sample the video clips at the rate of one frame for every second in each clip. Finally, we obtain 39,461 person images including 1581 person instances.

III-B Annotation

We utilize an annotation tool to label the captured video images. The whole annotating process is divided into three steps. The first step is object bounding box annotation, where all visible persons are manually marked with their positions in the images. With the bounding box positions, our dataset can also served as an aerial person detection and tracking benchmark dataset. The annotated product of this step is the smallest rectangle bounding box containing the complete person, which can be stored automatically.

The second step is cross-camera person matching and person ID Assignment, which is the most time-consuming step in the entire process. The same person appearing in two video clips need to be grouped together by manually searching and grouping. The same person instances are assigned with an unique number, ranging from $0\sim 1580$. Then the annotation files which contain the position of person instances and their ID numbers are generated automatically for each group. Finally, we crop the person instances from the original images to get the aerial person ReID dataset according to the information provided by the generated annotation files.

In addition, to more comprehensively illustrate our dataset, we compare our PRAI-1581 dataset with the Market1501 [5], MRP [29] and AVI [26] dataset from six aspects, including No. of images, IDs, and UAVs, the camera setting platform, height, and the situation of the camera. As can be seen from Table I, Market-1501 [5] is a commonly used ReID dataset under traditional surveillance settings. MRP [29] is the first ReID dataset captured by UAVs while its purpose is to verify the ReID algorithms on mobile platform, so the settings of the flight are not facing to the practical UAV surveillance scenarios. AVI [26] is a dataset for violent activity identification which is also captured by UAVs. Compared to the above described datasets, our PRAI-1581 is the first person ReID dataset facing to the practical UAV surveillance scenarios. To visually compare the four datasets, we show some example images of each dataset in Figure 2.

III-C Features of Our Dataset

Captured by UAVs flying at rich altitudes.

To increase the diversity of the images, the UAVs are controlled to fly at different altitudes, ranging from 20 to 60 meters across an outdoor environment. Figure 5 shows person images which are captured at different altitude, and it can be seen that our dataset has rich scale diversity, which is much more challenging than single scale persons in traditionally person ReID task with fixed camera emplacement. Figure 4 shows the sample distribution over the UAV flying height for our dataset.

Low resolution.

As shown in Figure 6, the size of each holistic image is $4K\times 2K$, while the resolution of the captured person is very low, ranging from about 30 to 150 pixels, which is far lower than traditional ReID person images. Low resolution is an especially challenging factor in aerial person ReID task. The statistical distribution over resolution (number of pixels) for our dataset is shown in Figure 3. It can be clearly seen that the average resolution is far lower than that in Market-1501.

Occlusions.

Severe and multi-type occlusions is another challenging factor in our dataset. Figure 6 shows some examples with different occlusions. In our dataset, there are about $20\%$ sample images are severely occluded typically by umbrellas, trees, shadows, and other persons etc.. Figure 7 shows the sample distribution over occlusion types in our dataset.

Diverse views and poses.

Figure 8 shows some images for one same person captured by UAVs under different perspectives. Our PRAI-1581 dataset includes diverse views and poses, including various profile views and top views, as UAVs can dynamically adapt the viewing positions and directions, while without being constrained by the fixed locations. So our dataset can meet the research need of recognition or matching related tasks for diverse angles, views and poses under complex conditions.

Due to the rich characteristics described above, our dataset can not only be served as a benchmark dataset for aerial person ReID facing to practical UAV surveillance scenarios, but also be fully tested for perception problem at a distance while aiming at building scale, pose, view (especially across profile view and top view), and occlusion invariant feature representations.

IV Approach

In this section, we elaborate the ReID algorithm based on subspace pooling, which achieved state-of-the-art performance on our dataset compared with many baseline methods. The framework of the proposed approach is shown in Figure  9. The subspace pooling layer is integrated with a backbone CNN model, and the architecture is fully differentiable so it can be optimized via standard back-propagation algorithm. Firstly, to emphasize the difference with the SVDNet [22], we briefly revisit the SVDNet method.

IV-A SVDNet Revisit

SVDNet makes an assumption that the featuring person descriptor in the last fully connected layer remains redundant and correlated. To eliminate the correlation, they adopt the SVD technique to decompose the weight matrix of the last fully-connected layer. As the weight matrix can be viewed as the projection basis, replacing the original weight matrix by using the left singular vectors weighted by the singular values ensures the orthogonality of the projection basis, thus leading to non-correlated feature descriptors in the fully-connected layer. Through the Restraint and Relaxation Iterative (RRI) training scheme, the orthogonality constraint can be integrated with the CNN training process. Note that the RRI training scheme is not an end-to-end way, CNN training and SVD decomposition are iteratively performed until convergence. On the other hand, we utilize subspace pooling to project the original CNN features into an orthogonal matrix while keeping the whole framework end-to-end trainable.

IV-B Subspace Pooling

In order to achieve the goal of reducing the feature redundancy and obtaining compact feature descriptors, we utilize the subspace pooling layer [24] combined with the backbone CNN to get the final feature representation. Specifically, denote $\mathbf{A}\in\mathbb{R}^{c\times hw}$ as the last convolution feature maps of the backbone CNN model, where $c$, $h$, $w$ represents channel, height and width of the feature maps respectively. Each row of $\mathbf{A}$ corresponds to one feature map enrolled into a $1$-D vector. Each column of $\mathbf{A}$ represents one $c$-D local feature vector at a specified spatial location. Generally, the SVD result of the CNN features can be formulated as

[TABLE]

Without loss of generality, we assume that the singular values and vectors are arranged as the descending order of $\sigma$. In many cases, the sum of the top 10% or even 1% of the singular values accounts for 99% of the sum of all the singular values. Thus we can approximately describe the matrix with the largest $k$ ($k\ll c$) singular values and the corresponding left singular vectors and right singular vectors. In addition, it is easy to prove that $\mathbf{A}_{k}=\sum_{i=1}^{k}{\sigma_{i}\mathbf{u}_{i}\mathbf{v}_{i}^{T}}$, is the best rank $k$ matrix approximations under the metric of Frobenius norm [24].

The output of the subspace pooling layer can be formulated as,

[TABLE]

That is, the subspace pooling layer transforms the input convolution feature maps into its $k$ left singular vectors corresponding to the largest $k$ singular values. The obtained orthogonal matrix geometrically supports a subspace of the Euclidean space.

IV-C Loss Function

In practice, to further improve the efficiency, a dimension reduction layer is introduced before we feed the convolution feature maps into the subspace pooling layer. Specifically, 1$\times$1 conv layer is utilized to reduce the feature dimensionality. After the subspace pooling layer, a linear classification layer is incorporated to project the feature vector to the person IDs.

We use two types of loss functions, softmax with cross entropy loss and triplet loss. The softmax with cross entropy loss can be formulated as follows:

[TABLE]

Where $N$ is the number of samples, $T$ is the number of target person IDs, $y$ is the ground truth label, $p$ is the predicted label after the softmax activation function.

To take full advantage of the hard triplet sampling mechanism, the batch hard triplet loss is adopted, which can be formulated as,

[TABLE]

Where $P\times K$ is the mini-batch size, sampling with P person IDs and K instances for each ID. $D_{a,p}^{*}$ is the largest distance among those between anchor sample and positive samples, and $D_{a,n}^{*}$ is the smallest distance among those between anchor sample and negative samples. $m$ is a hyper-parameter which denotes the margin in triplet loss. In order to verify the effectiveness of subspace pooling layer, we conduct extensive experiments in Section V on our collected dataset with many baselines and the proposed method achieves state-of-the-art performance compared with the baseline methods. Detailed results are shown in Section V.

IV-D Connections Between Subspace Pooling and SVDNet.

Although the subspace pooling layer and SVDNet both are benefited from the orthogonal projection or SVD technique, they differs in two major aspects. Firstly, SVDNet performs SVD on the weight vectors within the last FC layer, while subspace pooling performs SVD on the CNN features maps which are projected into a Riemannian manifold [25]. Secondly, their optimization schemes are totally different. Restraint and Relaxation Iteration (RRI) is the key procedure in training SVDNet, which is involved with iterative SVD decomposition and CNN training steps. However, The parameters of the whole model in subspace pooling method are learned in an end-to-end manner, which makes the training method more efficient and effective. Thus, although both subspace pooling and SVDNet establish a connection between CNN and SVD, their network models and operating mechanisms are totally different. In the experiments, we show that the subspace pooling method outperforms the SVDNet by a large margin on our dataset.

V Experiments

In this section, we empirically evaluate many baseline methods and representative state-of-the-art person ReID algorithms on our collected PRAI-1581 datasets, then the proposed subspace pooling method is reported and compared. Firstly, for fair comparison, we split the dataset into training and test set for only once and fix the partition of dataset for all the experiments. Then, we introduce the implementation details and evaluation metrics in Section V-B. The results of the baseline and state-of-the-art methods are reported in Section V-C. And the proposed subspace pooling method is evaluated in Section V-D. Lastly, we make some ablation studies in Section V-E

V-A Dataset Partition

Our PRAI-1581 dataset contains a total of 39,461 images for 1581 individual identities. We randomly divide the dataset into training and test set for only once and keep the partition fixed in all the experiments for fair comparison. The ratio of training and testing set is roughly set to 1:1. The training set includes 19,523 images of 782 identities. And the rest of the dataset is used as the test set with 19,938 images of 799 identities. To be more specific, the numbers of query and gallery images are 4680 and 15,258 respectively.

V-B Implementation Details and Evaluation Metrics

In our experiments, we use ResNet-50 as the backbone CNN model as it is a very popular model for the ReID task. The parameters of the backbone model are pre-trained on the ImageNet-2012 dataset. The batchsize is set to $32\times 4=128$, sampling with 32 different identities and 4 instances per identity in each mini-batch. The training process takes 300 epochs in total. The Adam optimizer with an initial learning rate of $2\times 10^{-4}$ is used, and learning rate is decayed at the $151^{th}$ epoch according to the exponential decay rule. The method is implemented based on the Pytorch platform and tested on a single NVIDIA 1080Ti GPU card.

For the evaluation of the experiments, we use the most common metrics in the field of person ReID, mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) at rank-1. In addition, when evaluating the proposed subspace pooling method, we use the F-score as an additional metric to indicate its practical performance.

V-C Baseline and State-of-the-art ReID Methods on PRAI-1581 Dataset

We report the performances of some typical methods and representative state-of-the-art person ReID methods on our dataset in Table II. For all the baseline methods, ResNet-50 is adopted as the backbone CNN model. Firstly, we enforce the identification loss (denoted as ID) and batch-hard triplet loss [6] (denoted as TL) on top of the backbone model respectively. The mAP and rank-1 accuracy of the ID method is 31.47% and 42.62% on our dataset, while the TL method achieves a mAP of 36.49% and a rank-1 accuracy of 47.47%, which shows that the batch-hard triplet loss performs better on our dataset for aerial person ReID.

Then, we test the Part-based Convolutional Baseline (PCB) which is proposed in [46] as a strong person ReID baseline. Note that the PCB method conducts uniform partition on the convolution feature maps for learning part-level features and it achieves a mAP of 37.15% and a rank-1 accuracy of 47.47%, which outperforms the naive ID method by +5.68% on mAP and +4.85% on rank-1. Similarly, a part based strong triplet loss (STL) method achieves comparable performance on our dataset, with mAP of 37.13% and rank-1 accuracy of 47.49%.

Lastly, we report representative state-of-the-art person ReID algorithms SVDNet [22], AlignedReID [47], Refined Part Pooling (RPP) [46], MBC [48], 2Stream [49], Part-align [50], DSR [51], IDE [52], DCGAN [28], Deep Embedding [32], MGN [53], OSNET [54] methods. The SVDNet also incorporates the SVD technique to reduce the redundancy of the person features and achieves a mAP of 36.7% and a rank-1 accuracy of 46.1%. AlignedReID learns to align the part features during the training process and it achieves a mAP of 37.64% and a rank-1 accuracy of 48.54%. Based on the PCB, RPP re-assigns the outliers in the evenly partitioned initial part to enhance the within part consistency and achieves a mAP of 38.45% and a rank-1 accuracy of 48.07%. OSNET [54] and MGN [53] which fuse multi-scale or multi-granularity global and part features, achieved state-of-the-art performance on our dataset. Note that although the part-based methods performs much better than the plain counterpart baselines for ReID in traditional surveillance cameras, they are not as effective as on our dataset probably because the diverse angles and views in aerial person images makes that the upright assumption of person images in traditional surveillance cameras can not hold, thus leading to bad part alignment results.

V-D The Subspace Pooling Method on PRAI-1581 Dataset

We then compare the proposed subspace pooling (SP) method with baselines on our PRAI-1581 dataset and the results are shown in Table III. It can be clearly seen that SP combined with ID method can improve the mAP from 31.47% to 37.88%, with 6.41% gains and improve the rank-1 accuracy from 42.46% to 48.33%, with 5.87% gains. While the SP combined with TL method can improve the mAP from 36.49% to 39.58%, with 3.09% gains and improve the rank-1 accuracy from 47.47% to 49.79%, with 2.32% gains. Note that the “SP+TL” method achieves state-of-the-art performance on our dataset, as it not only outperforms the baseline methods by a large margin, but also outperforms state-of-the-art person ReID algorithms such as SVDNet [22], AlignedReID [47] and “PCB+RPP” [46].

In addition, we qualitatively show some experimental results of the proposed method in Figure 10 and Figure 11. The leftmost column shows some query samples and the top-10 images in the ranking list of the gallery images are listed after the queries. In the Figures, images with Green boxes indicate the true matches to the query image, and the red boxes illustrate the false matches to the query image. Figure 10 shows some successful cases of our method, and Figure 11 gives some failure cases. It can be seen from Figure 11 that the main reasons for the failure cases are due to occlusion (1st row), low resolution (2nd row), view angle and shadows (3rd row), illumination and shade (4th row).

V-E Discussion

Dataset Partition.

To fully evaluate the proposed method on our dataset, we tried different partition ratios of training over test set and the results are reported in Table IV. Generally, more training samples can get better performance. It can also be inferred from Table IV that the ReID task in aerial views is much more challenging than traditional ReID with fixed camera emplacement.

Backbone Networks.

We further employ different backbone networks to conduct a comprehensive performance evaluation and comparison of the proposed method on our dataset. The results are reported in Table V. Note that when DenseNet-161 is adopted as the backbone network, it achieves better performance. However, DenseNet-161 takes much more computing and memory resources.

Results on Other Datasets.

To further verify the effectiveness of the proposed method, we compare the proposed subspace pooling method and its baseline on Market-1501, DukeMTMC and CUHK-03 datasets. The results of the experiments are all based on ResNet-50 backbone network. As can be seen from Table VI, the baseline method achieves 71.44% mAP and 86.49% Rank-1 accuracy on Market-1501. With the help of the SP layer, the mAP is increased to 72.51% and the rank-1 accuracy is increased to 86.52%. On DukeMTMC dataset, the mAP and Rank-1 accuracy of the baseline method is 60.83% and 77.51% respectively. While our method increases the mAP to 63.88%, with 3.05% gains and Rank-1 accuracy to 79.35%, with 1.84% gains, respectively. Table VI also gives the comparison results on CUHK03 dataset. The mAP of the baseline method is 49.36% and the rank-1 accuracy is 53.93%, our method achieves 51.11% mAP and 55.86% rank-1 accuracy. Note that our method are just based on global features while the state-of-the-art methods on traditional ReID datasets mostly include adequate local feature alignment. Despite of this, our subspace pooling method improves the baselines on all the three datasets.

Hyper-parameters.

To further investigate how some hyper-parameters affect the final ReID performance, we study the effect of $1\times 1$ convolution layer before the subspace pooling layer. The original number of feature maps before feeding into $1\times 1$ convolution layer is 2048. To get a trade-off between efficiency and the accuracy, we set the output of the $1\times 1$ convolution layer to 512. The results of different $1\times 1$ convolution layer output sizes are shown in Table VII.

VI Conclusion

In this paper we collect a large scale airborne person ReID dataset which consists of 39,461 images of 1581 person identities. The images of the dataset are shot by two DJI consumer UAVs flying at an altitude ranging from 20 to 60 meters above the ground, which covers most of the real UAV surveillance scenarios. To the best of our knowledge, it is the first large person ReID dataset in aerial imagery facing to the real UAV surveillance scenarios. We hope that our dataset can facilitate the research area of person ReID in aerial imagery or even person ReID on mobile platform. In addition, we propose to utilize subspace pooling of convolution feature maps to learn a discriminative and compact feature descriptor for ReID in aerial imagery and more importantly, it can be trained via an end-to-end fashion effectively. Extensive experiments show that our method performs favorably against state of the arts on our dataset.