A Deep Four-Stream Siamese Convolutional Neural Network with Joint Verification and Identification Loss for Person Re-detection

TL;DR

This paper introduces a four-stream Siamese CNN for person re-identification that jointly optimizes verification and identification losses, outperforming triplet-based methods and achieving state-of-the-art results on multiple datasets.

Contribution

The paper presents a novel four-image group approach with joint loss optimization, improving generalization and discriminative feature learning in person re-identification.

Findings

Achieves state-of-the-art performance on VIPeR, CUHK01, CUHK03, and PRID2011 datasets.

Outperforms triplet-based deep networks in person re-identification tasks.

Effectively reduces intra-class variation and increases inter-class variation.

Abstract

State-of-the-art person re-identification systems that employ a triplet based deep network suffer from a poor generalization capability. In this paper, we propose a four stream Siamese deep convolutional neural network for person redetection that jointly optimises verification and identification losses over a four image input group. Specifically, the proposed method overcomes the weakness of the typical triplet formulation by using groups of four images featuring two matched (i.e. the same identity) and two mismatched images. This allows us to jointly increase the interclass variations and reduce the intra-class variations in the learned feature space. The proposed approach also optimises over both the identification and verification losses, further minimising intra-class variation and maximising inter-class variation, improving overall performance. Extensive experiments on four challenging datasets, VIPeR, CUHK01, CUHK03 and PRID2011, demonstrates that the proposed approach achieves state-of-the-art performance.

Tables

Table 1. Table 1: Results of our proposed method in comparison with other state-of-the-art approaches on the VIPeR [ 12 ] dataset. In the type column, ID indicates an identification loss is used, V that a verification loss is used, and ID + V indicates that a combined loss is used.

Method	Type	Rank1	Rank5	Rank10
DML [44]	ID	28.23	59.27	73.45
FLCA [27]	ID	42.5	72	91.72
DGD [42]	ID	38.6	-	-
Improved Trp [6]	V	47.8	74.7	84.8
SIRCIR [37]	ID+V	35.76	-	-
GSCNN [35]	ID	37.8	66.9	77.4
DRPR [3]	V	38.37	69.22	81.33
DRDC [8]	V	40.5	60.8	70.4
IDLA [1]	ID	34.81	63.32	74.79
EDM [32]	ID	40.91	-	-
MTDNet [5]	ID+V	45.89	71.84	83.23
BTL [4]	V	49.05	73.10	81.96
Spindle Net [45]	ID	53.8	74.1	83.2
The proposed method	V	62.5	83.1	92.4
The proposed method	ID	58.2	76.7	85.8
The proposed method	ID+V	68.7	88.9	94.6

Table 2. Table 2: Results of our proposed method in comparison with other state-of-the-art approaches on CUHK03 [ 20 ] dataset. Type column is as per Table 1.

Method	Type	Rank1	Rank5	Rank10
PersonNet [41]	V	64.8	89.4	94.92
FPNN [20]	ID	20.65	51.32	68.74
CAN [24]	ID	72.3	93.8	98.4
DGD [42]	ID	75.3	-	-
SIRCIR [37]	ID+V	52.17	-	-
DLCNN [50]	ID+V	83.4	97.1	98.7
GSCNN [35]	ID	68.1	88.1	94.6
IDLA [1]	ID	54.74	-	-
EDM [32]	ID	61.32	-	-
MTDNet [5]	ID+V	74.68	95.9	97.47
BTL [4]	V	75.53	95.15	99.16
Spindle Net [45]	ID	88.5	97.8	98.6
The proposed method	V	81.14	95.6	98.7
The proposed method	ID	76.9	91.3	97.4
The proposed method	ID+V	85.5	98.74	99.8

Table 3. Table 3: Results of our proposed method in comparison with other state-of-the-art approaches on CUHK01 [ 19 ] dataset. Type column is as per Table 1.

Method	Type	Rank1	Rank5	Rank10
FPNN [20]	ID	27.87	-	-
FLCA [27]	ID	46.8	71.8	80.5
DGD [42]	ID	71.7	88.6	92.6
Improved Trp [6]	V	53.7	84.3	91
SIRCIR [37]	ID+V	71.8	-	-
DRPR [3]	V	70.94	92.3	96.9
IDLA [1]	ID	65	89.5	93
PersonNet [41]	V	71.14	90	95
EDM [32]	ID	69.38	-	-
MTDNet [5]	ID+V	77.5	95.0	97.5
BTL [4]	V	62.55	83.44	89.71
Spindle Net [45]	ID	79.9	94.4	97.1
The proposed method	V	80.12	95.0	97.23
The proposed method	ID	78.01	93.4	96.8
The proposed method	ID+V	83.95	98.15	98.97

Table 4. Table 4: Results of our proposed method in comparison with other state-of-the-art approaches on PRID2011 [ 14 ] dataset. Type column is as per Table 1.

Method	Type	Rank1	Rank5	Rank10
DML [44]	ID	17.9	37.5	45.9
DGD [42]	ID	64	-	-
Improved Trp [6]	V	22	-	47
MTDNet [5]	ID+V	32.0	51.0	62.0
The proposed method	V	69	90	95
The proposed method	ID	66	87	91
The proposed method	ID+V	75	93	97

Table 5. Table 5: Results of our proposed method in comparison with other state-of-the-art approaches on VIPeR [ 12 ] , CUHK03 [ 20 ] , CUHK01 [ 19 ] and PRID2011 [ 14 ] dataset.

Database	Loss	Rank1	Rank5	Rank10
VIPeR	Triplet	61	92	95
	Quartet	68.7	88.9	94.6
CUHK03	Triplet	83.8	97.8	98.8
	Quartet	85.5	98.74	99.8
CUHK01	Triplet	79.5	93	94.5
	Quartet	83.95	98.15	98.97
PRID2011	Triplet	71	91	96
	Quartet	75	93	97

Equations

$$\parallel {\Theta }_w(A_i^1)-{\Theta }_w(A_i^2){\parallel }^2<\parallel {\Theta }_w(A_i^1)-{\Theta }_w(A_i^3){\parallel }^2.$$

$$\displaystyle Loss_{triplet}=\sum_{i=1}^{n}\Big{(}max\big{\{}\parallel\Theta_{w}(A_{i}^{1})-\Theta_{w}(A_{i}^{2})\parallel^{2}$$

$$\displaystyle-\parallel\Theta_{w}(A_{i}^{1})-\Theta_{w}(A_{i}^{3})\parallel^{2},margin\big{\}}\Big{)}.$$

$$\displaystyle Loss_{quartet}=\sum_{i=1}^{n}\Big{(}max\big{\{}\parallel\Theta_{w}(A_{i}^{1})-\Theta_{w}(A_{i}^{2})\parallel^{2}$$

$$-\parallel {\Theta }_w(A_i^1)-{\Theta }_w(A_i^3){\parallel }^2+\parallel {\Theta }_w(A_i^1)-{\Theta }_w(A_i^2){\parallel }^2$$

$$\displaystyle-\parallel\Theta_{w}(A_{i}^{4})-\Theta_{w}(A_{i}^{3})\parallel^{2},margin\big{\}}\Big{)}.$$

$$\frac{\partial d(A_i,w)}{\partial w}=2({\Theta }_w(A_i^1)-{\Theta }_w(A_i^2))\frac{\partial {\Theta }_w(A_i^1)-\partial {\Theta }_w(A_i^2)}{\partial w}$$

$$-2({\Theta }_w(A_i^1)-{\Theta }_w(A_i^3))\frac{\partial {\Theta }_w(A_i^1)-\partial {\Theta }_w(A_i^3)}{\partial w}$$

$$+2({\Theta }_w(A_i^1)-{\Theta }_w(A_i^2))\frac{\partial {\Theta }_w(A_i^1)-\partial {\Theta }_w(A_i^2)}{\partial w}$$

$$-2({\Theta }_w(A_i^4)-{\Theta }_w(A_i^3))\frac{\partial {\Theta }_w(A_i^4)-\partial {\Theta }_w(A_i^3)}{\partial w}.$$

$$Los{s}_{identification}=-\sum _{i=1}^n{p}_{i}log\overline{p_i}=-log{\bar{p}}_t,$$

$$CMC(i)=\sum _{r=1}^{i}q(r),$$

Full text

1 Introduction

Person re-detection is the task of matching pedestrians across different spatial and temporal locations over multiple cameras (see Figure 1). For instance, determining whether the person observed at point A is the same person now visible at point B. Despite the efforts of researchers, it remains an unsolved problem, with the most challenging issue being to adapt to large changes in appearance caused by variations in lighting, and changes in the pose of the subject and the camera. Existing research on person re-detection has mainly focused extracting better features from the input images by developing feature extraction methods that are invariant to changes in condition; developing distance metrics to compare pairs of extracted features; or both.

Over the past few years, deep convolutional neural network (DCNN) methods have shown their potential with significant performance improvements in person re-detection. Some researchers [20, 1, 37] have considered person re-detection as a multi class recognition problem and classification loss is used to train the network; while others consider it a verification problem [3, 6, 8, 37].

For verification, most researchers adopted either a Siamese network or a triplet based network. Siamese architectures take a doublet (pair of images) as input and pull the images of the same person close together in feature space, while features extracted from different people will be kept separate from each other in the feature space.

Usually, verification or similarity regression models take a pair of images as input for training purposes, which are then used to extract identifiable features to determine whether the query images (person-of-interest) are of the same person or not. On the contrary, identification or multi class recognition models classify an input image into a large number of identity classes to predict the identity of the input image. Verification and identification models are shown in Figure 2 and Figure 3, respectively. With regards to feature extraction, both approaches are different and have their own strengths and weaknesses.

The two approaches also differ in that an identification approach necessitates a closed-world view, while the verification approach allows an open-world view. The identification methods assume that the identities of subjects to be recognized exist in the gallery, and subjects outside this list cannot be identified. By contrast, a verification task considers person re-detection as an open world scenario by using a distance measure and threshold to determine if two images belong to the same person or not. On the other hand, triplet based models takes three images as input: an anchor image, a positive image and a negative image; and the network enforces that the distance between the anchor and positive image should be less than the distance between the anchor and negative image. Essentially, the overall network architecture is a Siamese network with either two or three branches for the pairwise and triplet loss, respectively. However, the triplet loss forces that the distance of intra-class identity to be less than the distance of inter-class identities only in cases where the test images are from the same identity. In a real world scenario, test subjects and their images are totally unseen, thus the triplet based framework suffers from a poor generalization capability.

The drawback of the verification model is that it does not consider all the information in the training data. This model considers only doublet (pairwise) or triplet labels of the relationship between a pair of images instead of other images in the dataset [34]. An identification model overcomes this by extracting non-linear features directly from the input image to learn the person ID (identification) during training, and the loss function is used in the final layer makes full use of the re-ID labels. However, when used outside of training, to overcome the closed-world nature of the training approach, identification methods extract features from the last fully connected layer and compare these, however there is no guarantee that features from the same subject will lie close to one another in the feature space. As can be seen from the above, the two approaches have contrasting strengths where an identification approach will increase inter-class variations and a verification approach will minimise the intra-class variations.

Motivated by the above observations, we propose to introduce a new loss function for person re-detection which we term the quartet loss function. This loss function enforces that the distance between the matched image pair is less than the distance between the mismatched image pair with respect to the same probe image, while simultaneously enforcing that the distance between positive pairs is less than the distance between negative pairs with respect to different probe images, resulting in better generalization and better performance. The proposed approach also combines elements of the verification and identification models to take full advantage of both approaches to maximize the performance of the person re-detection system. To summarize, our contributions are:

• •

We introduce a new loss function called the quartet loss to train the network where four images are taken as input, improving the generalization capability of the network, thus the trained network is able to differentiate between the positive and negative pairs not only with the same probe image, but also with respect to different probe images.

• •

We propose a unified deep learning person re-detection network that jointly optimizes both identification (who is this person?) and verification tasks (are these images of the same person?) to maximize their strengths, thus improving person re-identification accuracy.

• •

We report a thorough experimental validation on the proposed losses in order to show how each performs separately i.e. only verification vs only identification vs identification and verification, to justify that the fusion of identification and verification helps to improve the performance.

• •

We report state-of-the-art accuracy compared to existing methods on four challenging person re-detection datasets: VIPeR [12], CUHK01 [19], CUHK03 [20] and PRID2011 [14].

The rest of the paper is organized as follows: Section 2 outlines existing research relating to person re-detection; Section 3 describes our proposed methodology; Section 4 presents our experimental setup and results; and Section 5 concludes the paper.

2 Related research

For person re-detection, several hand-crafted feature methods such as colour histograms [11, 10, 48, 43, 46, 47, 22] symmetry-driven accumulation of local features (SDALF) [9], adaboost [12], local binary pattern (LBP) [19, 18, 26, 43], gabor features [19, 49], local maximum occurrence (LOMO) [23, 22], speeded up robust features (SURF) [2], spatiotemporal HOGs (STHOGs) [16], and scale-invariant feature transform (SIFT) [25] have been used for feature extraction. Once the features are extracted, a distance metric such as euclidean distance [7], Bhattacharyya coefficients [7], large margin nearest neighbour (LMNN) [40], relative distance comparison (RDC) [49], probabilistic relative distance comparison (PRDC) [48], local fisher discriminant analysis (LFDA) [33, 29], Mahalanobis distance metric [31], or locally-adaptive decision function (LADF) [21] is considered to compare the extracted features for person re-identification. However, handcrafted features are not discriminative enough, and in particular they are not reliable and invariant against changes in pose, viewpoint, illumination and scene occlusion.

Deep convolutional neural networks (DCNNs) have shown tremendous potential in a variety of pattern recognition and machine learning tasks [17, 30]. By learning feature representations directly from the data, DCNNs have demonstrated their power in complex and varied image and computer vision tasks, and have shown an ability to uncover complex relationships within the data. Some researchers considered DCNNs for person re-detection and the performance has been shown to be comparatively better than using handcrafted features.

The approaches of [20, 1, 37, 36, 35, 44] adopted a Siamese architecture for person re-detection and treated the task as a classification problem. The core concept of the Siamese Convolutional Neural Network (SCNN) architecture is to build a system where images belonging to the same identity will be close to each other in the feature space, whereas images from different identities will well separated. Although [44] first proposed the Siamese based classification model for person re-detection, [20] matched the filter responses of local patches of images by introducing a filter pairing neural network and [1] presented a new layer to compute the neighbourhood difference between two input images. Varior et al. [35] introduced a gating function with the Siamese architecture to capture the subtle structures of a pair of probe images. This gate function was used after each convolution layer. The limitation of this method is that the probe image has to be paired with each gallery image before being fed into the network which is inefficient when dealing with large datasets. Similar to [35], [24] proposed a Siamese architecture and adopted a soft attention based model to focus on the local body parts of an input image pair instead of global body parts. Another study [28] introduced a video-based re-detection system where the SCNN was used for feature extraction followed by a recurrent neural network (RNN) layer to transfer information within time-steps. In [38], triplet units were adopted to train the network with the triplet loss function in order to learn image similarity metrics. In [8], a triplet network was used for person re-detection where only ranking was taken into consideration. The triplet loss function was improved in [6] by inserting a new term within the original triplet loss function and achieved state-of-the-art performance, however they considered person re-detection as a verification task. In [4], a new loss function was introduced where a four image input is taken into consideration for training, where two margins (similar to [6]) are used. The function of the first margin is the same as the traditional triplet loss while the second margin is used to further maximize the inter-class distance. As the second margin is weaker than the first margin, in their ranking loss the triplet loss dominates which forces to the network minimize the intra-class distance only in cases where the test images are from the same identity. By contrast, in our proposed loss, we equally consider minimizing the intra-class distance in the case of either having the same or different probe images. Furthermore, [4] considered person re-detection as a verification task while we consider it jointly as a verification and identification task.

Form the above analysis, it is noted that most of the research adopted either pairwise or triplet verification loss for person re-detection, although the overall network is typically a Siamese based architecture. Although [5] considered person re-detection jointly as a ranking and classification problem, they adopted the traditional triplet loss which forces that the distance of intra-class identities to be less than the distance of inter-class identities only in cases where the test images are from the same identity. Wang et al. [37] also considered person re-detection as both verification and identification tasks, however, they trained the two losses separately and fuse these two tasks only at the score level. Thus the trained network is not be able to learn to perform both tasks in a single framework.

Inspired by the success of deep convolutional neural networks for person re-detection, we propose a deep four stream convolutional architecture that differs from the above deep learning based methods in network architecture and loss function. Specifically, we introduce a unified deep network with quartet loss that simultaneously optimises for the identification and verification tasks in order to re-detect the target person. By taking a group of four images as a training unit we can achieve better re-detection performance as the network is better equipped to increase the inter-class distance while decreasing the intra-class distance with respect to both the same probe image (similar to the original triplet loss) and a different probe image (our proposed quartet loss) simultaneously. Furthermore, we provide a comparison of the joint verification and identification tasks with only verification and only identification to show that learning the tasks jointly performs better than either one alone.

3 Proposed Methodology

In this paper, we consider person re-detection jointly as an identification and verification task, in contrast to other approaches which consider it either as a multi class recognition (identification) task or a similarity regression (verification) task. The verification task considers person re-detection as a binary class recognition problem which determines whether the probe image belongs to the same person or not while the identification model is a multi-class recognition task, which predicts the identity of the probe image. These two tasks are combined in our model to re-detect people with increased accuracy.

3.1 The Overall Framework

Our network is a four stream Siamese convolutional neural network with a quartet loss function that combines the verification and identification losses. We use a group of four images as input for training purposes as illustrated in Figure 4, followed by a quartet loss and a softmax loss function for verification and identification tasks, respectively. In this paper, the input selection is denoted as, $A_{i}$ = $A_{i}^{1}$, $A_{i}^{2}$, $A_{i}^{3}$, $A_{i}^{4}$ where $A_{i}^{1}$ is the anchor image, $A_{i}^{2}$ is the positive image (i.e. $A_{i}^{1}$ and $A_{i}^{2}$ are the images belonging to the same person) while $A_{i}^{3}$ and $A_{i}^{4}$ are the negative images (i.e. images from two different people, both different to the person in the positive images). The network consists of four ImageNet [17] pre-trained CNN models and two losses. The four images share the weights and biases through the four CNNs. After training, the CNN architecture is able to differentiate the matched and mismatched pairs through the learned feature space with respect to the same and different probe images. In the model, Alexnet is used as a pre-trained network, thus we follow the same architecture as Alexnet, consisting of five convolution layers and three fully connected layers. From [39], the lower layers of deep architectures encode more discriminative features to capture intra-class variations and provide more detailed local features, whereas higher layers capture semantic features. Thus for verification, we compare the images based on low level features. For identification, features are extracted from higher layers as these features focus on identifiable local semantic concepts.

3.1.1 Four Stream Siamese Based Verification and Identification Model with Quartet loss

Our proposed quartet loss is designed based on the triplet loss, thus, in this section we introduce the original triplet loss and then we present our proposed loss.

In the triplet network, three images are used as an input to train the network where for each positive pair, a third (negative) image is randomly selected to complete the group of three (i.e. input triplet $A_{i}$ = $A_{i}^{1}$, $A_{i}^{2}$, $A_{i}^{3}$, $A_{i}^{1}$ and $A_{i}^{2}$ are images from the same person and $A_{i}^{1}$ and $A_{i}^{3}$ belong to a different person). The learned feature space is expressed as, $\Theta_{w}(I)$ which is the feature representation for image $I$. The learned feature satisfies the following condition under Euclidean distance or $L_{2}$ norm,

[TABLE]

In other words, in the learned feature space the distance between a negative pair of images should be greater than the distance between a positive pair. If three images are taken as input, the triplet loss function can be formulated as,

[TABLE]

Thus, the triplet loss enforces that the distance between the matched pair should be smaller than the distance between the mismatched pair by a predefined margin which enables the network to differentiate positive and negative samples when the probe image belongs to the same person. The max operation is used with the margin to prevent the overall value of the objective function from being dominated by easily identifiable triplets, the same technique as in hinge-loss functions.

In our proposed quartet network, four images are taken as an input to train the network where for each positive pair, third and fourth (negative) images are randomly selected to complete the group of four (i.e. input unit $A_{i}$ = $A_{i}^{1}$, $A_{i}^{2}$, $A_{i}^{3}$, $A_{i}^{4}$, $A_{i}^{1}$ and $A_{i}^{2}$ are images from the same person and $A_{i}^{3}$ and $A_{i}^{4}$ belong to different people). All the images are resized to 227 x 227 and fed into four DCNNs which share the same weights and biases, thus the feature extraction will be consistent for each image. The proposed quartet loss can be expressed as,

[TABLE]

In other words, the quartet loss increases the inter-class distance over the intra-class distance with the same target image as well as different target images, improving the overall performance. The positive pair is comprised of $\Theta_{w}(A_{i}^{1})$ and $\Theta_{w}(A_{i}^{2})$ and is included twice, to compensate for having two negative pairs (i.e. $\Theta_{w}(A_{i}^{1})$, $\Theta_{w}(A_{i}^{3})$; and $\Theta_{w}(A_{i}^{3})$, $\Theta_{w}(A_{i}^{4})$). The first of the negative pairs shares a common probe image with the positive pair (i.e. $\Theta_{w}(A_{i}^{1})$), while the second negative pair uses two different images. Thus, the proposed verification loss is able to maximize the inter-class distance even if the target image comes from a different identity. As the four image input is incremental as the dataset gets larger, a margin is set to restrict the distance between positive and negative pairs. Here the square of the distance is used to simplify the partial derivative calculation. Gradient descent is used to train the deep convolutional network and the gradient is calculated as partial derivatives of Equation (3) using the chain rule. The gradient is obtained as follows,

[TABLE]

The values for Equation (4) can be acquired by forward and back propagation for each input image in the training input unit. We note that not only does this formulation help improve learning, but the use of quartets helps generate additional training permutations of the data to actually train the network, helping to overcome the relatively small amount of training data typically available in this domain.

In the identification portion, to determine the identity of the probe images the network enforces images of different identities to be far apart in the learned feature space. As our training network consists of four images as an input, this leads to three pairs being formed for identification (one from images of the same identity and two from images of different identity). The identification loss is determined using a softmax layer to classify the similarity between the test images and the gallery images. Similar to traditional multi-class recognition approaches, we use the cross-entropy loss for identity prediction, which is defined as,

[TABLE]

where $p_{i}$ is the probability distribution of the target, $i$ is the number of classes, $\bar{p}_{i}$ is the predicted probability distribution and $t$ is the target class. In our case, we can define it simply as a binary cross-entropy loss as this is a cross entropy for the two-class case. Thus, the output will be 1 if the probe image is matched with the positive pair otherwise it will be 0 in case of a mismatched pair.

3.2 Network Architecture

Our network consists of five convolution layers as shown in Figure 4: the first two are used for verification and identification; and the remaining thre are used only for identification. For verification, features extracted from the first two convolutional layers are sent to a fully connected layer followed by the verification loss function. For the identification task, after the two convolutional layers, three pairs are formed as mentioned above (one comprising images of the same identify, two being image pairs with different identifies) which are sent to the three convolutional layers followed by a softmax layer to determine the similarity. In the test phase, the two input images are sent through all layers, where the last layer obtains the similarity probability of a test pair. These similarity scores are used to rank probe subjects similarly to each gallery image.

4 Experiments & Results

We conduct extensive experiments to evaluate the performance of the proposed method on four person re-detection datasets: VIPeR, CUHK03, CUHK01 and PRID2011.

4.1 Datasets

We used four challenging and popular person re-detection benchmark datasets: VIPeR, CUHK03, CUHK01, and PRID2011; for our experiments. All the datasets have images captured by different cameras. A brief explanation on four datasets are as follows:

• •

VIPeR [12] dataset: Consists of 632 identities, each of which has a pair of images captured with different cameras with distinct angles, illuminations and poses. In our experiment, 316 identities are randomly chosen for testing, leaving the rest for training, as per [6] .

• •

CUHK03 [20] dataset: Consists of 14097 images of 1467 identities collected from 6 surveillance cameras in the CUHK campus. In this dataset, each identity is taken from two disjoint camera views. We use the training and testing sets as proposed in [45].

• •

CUHK01 [19] dataset: Consists of 971 persons from two camera views. Each person has four images, two from each camera. We use the training and testing sets as proposed in [45].

• •

PRID2011 [14] dataset: The PRID2011 dataset extracts images from video recorded by static surveillance cameras with two camera views, each of which contains 385 and 749 identities. In this dataset, only 200 persons are seen in both camera views. We follow the same training and testing sets as [42].

4.2 Evaluation Protocol

We use cumulative matching characteristic (CMC) curves for performance evaluation, which are widely used for person re-detection evaluations, and determine rank-1, rank-5, and rank-10 accuracy. The CMC represents the probability of finding the correct match for the probe image in the top n matches where the best case is given by a re-detection rate of 100% at the rank-1, i.e. the correct match is always the highest rank. CMC can be defined as:

[TABLE]

where, $i$ represents the rank and $q(r)$ is the number of correct re-detected queries.

4.3 Experimental Setup

The proposed method is implemented using the Caffe framework [15]. All images are resized to 227x227 for Alexnet during training. The network is pre-trained on ImageNet [17]. We set the learning rate to 0.0001 throughout the experiments. Batch size is set to 128 and we train the network for 30,000 iterations with an NVIDIA GPU, which takes about 24-30 hours. We use stochastic gradient descent (SGD) to update the parameters of the network. Our results on four different datasets (VIPeR, CUHK03, CUHK01, and PRID2011) are given in Tables 1 to 4 respectively, alongside other state-of-the-art approaches on these datasets.

4.4 Comparisons with state-of-the-art approaches

As shown in Table 1, 2, 3 and 4, we compare the results of our proposed method with state-of-the-art approaches on four datasets (VIPeR, CUHK03, CUHK01, and PRID2011) in terms of rank-1, rank-5 and rank-10 accuracy. We report the single-query evaluation results. We also report the result of the proposed approach using only verification and only identification losses, which is compared with the proposed joint verification and identification task.

For the VIPeR dataset, our model achieves 68.7% rank-1 accuracy, outperforming state-of-the-art approaches, [6] and [45] by 20.9% and 14.9%, respectively. Cheng et al. [6] considered person re-detection as a verification task and inserted a new term within original triplet loss, and [45] adopted identification task for person re-detection and extracted features from different body regions.

For the CUHK03 dataset, the largest dataset for person re-detection consisting of 1467 identities, our proposed method achieves 85.5% rank-1 accuracy which outperforms most state-of-the-art approaches, such as, DLCNN [50] by 2.1% who considered person re-detection both as a verification and identification task. But in contrast to our proposed work they adopted a pairwise verification loss whereas we consider a quartet verification loss, thus improving the generalization capability. Although in terms of rank-1 accuracy, [45] beats our result by 3%, they considered human landmark information and were thus able to leverage more detailed information by dividing the body into seven regions for feature extraction. Thus with a larger dataset, they can extract more identifiable features from different regions. Despite this additional information however, our proposed method outperforms their method at rank-5 and rank-10 by 0.94% and 1.20% respectively.

For the CUHK01 dataset, the previous best methods are [37], [42] and [45] with 71.8% and 71.7% and 79.9% rank 1 accuracy respectively. In [37], both verification and identification tasks were adopted, but in contrast to our approach they simply join these two methods at the score level, whereas we combine the verification and identification tasks within the network architecture. In [42], Xiao et al. considered person re-detection as an Identification task where 6 datasets (VIPeR, CUHK01, PRID2011, i-LIDS, CUHK03, 3DPeS) were joined together for training purpose and [45] who extracted local features from seven body parts for re-detection. Notably, our rank-1 accuracy is 83.95% which is much better than above state-of-the-art methods.

For the PRID2011 dataset, the existing best result was achieved by [42], which as noted above used 6 datasets for training and considered person re-detection as an identification task. By contrast, our combined verification and identification model with the four stream Siamese network performs better than this and other approaches, achieving a 75% rank-1 accuracy.

From the above comparisons with other methods, it is worth noting that almost all the evaluated methods considered person re-detection either as an identification or verification problem alone except [37] and [5] who considered both tasks jointly, but [37] combined them at the score level and [5] adopted the traditional triplet loss function. In contrast, we fuse both tasks within the proposed network architecture, enabling the network to jointly learn to perform both with quartet loss function which can overcome the weakness of the triplet loss function by improving the generalization capability of the network. For completeness, we also evaluate the proposed approach using each loss on it’s own. From Tables 1 to 4, we can clearly see that the proposed approach outperforms the state of the art in all but one situation (compared to [44] on CUHK-03 at Rank-1), and the joint loss clearly outperforms either loss individually, clearly indicating that person re-detection performs better if it is considered both as a verification and identification task rather than considering it as either one independently. Furthermore, we propose a quartet loss method that improves the generalization capability by increasing the inter-class variation with reducing the intra-class variation through the use of different probe images. This is evidenced by the highly competitive results of the proposed approach with only a single loss. For instance in Table 1 for the VIPeR dataset, using only the verification or identification loss outperforms all baselines at Rank-1, 5 and 10.

4.5 Comparison Between Triplet and Quartet Losses

To evaluate the benefit of the quartet over the conventional triplet loss, we formulate the proposed joint verification and identification approach using a triplet of input images, and compare this to the complete proposed approach (quartet with joint verification and identification losses). Results are shown in Table 5. From this, we can clearly see that the proposed use of a quartet of input images leads to improved performance. Furthermore, comparing Table 5 to Tables 1, 2, 3 and 4, we can see that using the proposed dual losses with a triplet formulation is still able to outperform all other baselines on VIPeR and PRID2011, and is only bested by [45] on CUHK03 and CUHK01. This highlights the twin gains by using both the quartet of input images and the joint verification and identification losses.

5 Conclusion and Future work

In this paper, we propose a deep learning person re-detection method by joint identification and verification learning. Since identification is adept at predicting the identity of the target image, and verification is effective in determining the relationship between target and gallery images, we combine these two approaches together to maximize the advantages of both of these methods. We further extend existing architectures through the use of a quartet of input images which are incorporated into the verification and identification loss function. This enforces the intra-class distance to be larger than the inter-class distance with respect to multiple different probe images. Experimental results shows the performance of the joint identification and verification method outperforms state-of-the-art techniques on four datasets, VIPeR, CUHK03, CUHK01, and PRID2011. In the future, we will investigate other ways to improve the performance of person re-identification, such as dividing the human body into local body parts for better feature extraction. Furthermore, we will expand our work by adding deep transfer learning to more effectively adapt to unseen conditions. Finally, more recent architectures such as ResNet [13] will be investigated as an alternative feature extraction pipeline.

Acknowledgement

This research was supported by the Australian Research Council’s Linkage Project “Improving Productivity and Efficiency of Australian Airports” (140100282). The authors would also like to thank QUT High Performance Computing (HPC) for providing the computational resources for this research.