Adaptive Exploration for Unsupervised Person Re-Identification

TL;DR

This paper introduces an Adaptive Exploration method for unsupervised person re-identification that effectively addresses domain bias by dynamically selecting neighborhoods and balancing their influence, improving accuracy across datasets.

Contribution

The proposed method adaptively selects neighborhoods for each person image and incorporates a balance strategy, advancing unsupervised domain adaptation in person re-ID.

Findings

Effective in reducing domain bias in re-ID tasks

Improves accuracy on large-scale datasets

Works with target-only and domain adaptive protocols

Abstract

Due to domain bias, directly deploying a deep person re-identification (re-ID) model trained on one dataset often achieves considerably poor accuracy on another dataset. In this paper, we propose an Adaptive Exploration (AE) method to address the domain-shift problem for re-ID in an unsupervised manner. Specifically, in the target domain, the re-ID model is inducted to 1) maximize distances between all person images and 2) minimize distances between similar person images. In the first case, by treating each person image as an individual class, a non-parametric classifier with a feature memory is exploited to encourage person images to move far away from each other. In the second case, according to a similarity threshold, our method adaptively selects neighborhoods for each person image in the feature space. By treating these similar person images as the same class, the non-parametric classifier forces them to stay closer. However, a problem of the adaptive selection is that, when an image has too many neighborhoods, it is more likely to attract other images as its neighborhoods. As a result, a minority of images may select a large number of neighborhoods while a majority of images have only a few neighborhoods. To address this issue, we additionally integrate a balance strategy into the adaptive selection. We evaluate our methods with two protocols. The first one is called "target-only re-ID", in which only the unlabeled target data is used for training. The second one is called "domain adaptive re-ID", in which both the source data and the target data are used during training. Experimental results on large-scale re-ID datasets demonstrate the effectiveness of our method. Our code has been released at https://github.com/dyh127/Adaptive-Exploration-for-Unsupervised-Person-Re-Identification.

Tables

Table 1. Table 1. Comparison of our models under different settings on DukeMTMC-reID(Duke) and Market-1501(Market). Source-only : Model is only trained on the source dataset. Target-only : Model is only trained on the target dataset. Transfer : Model is trained on both source dataset and target dataset. Cam : CamStyle (Zhong et al . , 2019b ) data augmentation. Src. : Source domain.

Methods			Market-1501 (%)					DukeMTMC-reID (%)
			Src.	rank-1	rank-5	rank-10	mAP	Src.	rank-1	rank-5	rank-10	mAP
Source-only			Duke	46.4	63.7	70.6	20.0	Market	30.3	45.4	52.5	15.9
w/o Cam	Target-only	adaptive selection	N/A	35.4	54.2	63.3	14.4	N/A	37.2	53.5	60.6	20.5
		adaptive selection + balance		57.1	75.3	81.8	34.5		56.1	72.6	78.7	35.5
	Transfer	adaptive selection	Duke	49.6	66.0	73.0	22.4	Market	45.8	60.8	68.8	28.9
		adaptive selection + balance		67.3	79.6	83.7	40.8		60.1	76.3	82.3	41.8
w/ Cam	Target-only	adaptive selection	N/A	55.4	74.8	81.6	23.4	N/A	42.5	57.9	64.2	19.4
		adaptive selection + balance		77.5	89.8	93.4	54.0		63.2	75.4	79.4	39.0
	Transfer	adaptive selection	Duke	66.6	83.6	89.1	35.1	Market	59.4	72.5	78.3	36.2
		adaptive selection + balance		81.6	91.9	94.6	58.0		67.9	79.2	83.6	46.7

Table 2. Table 2. Comparison between triplet loss and contrastive loss

Methods	DukeMTMC-reID to Market-1501 (%)				Market-1501 to DukeMTMC-reID (%)
	rank-1	rank-5	rank-10	mAP	rank-1	rank-5	rank-10	mAP
Contrastive	48.1	64.3	70.8	20.7	35.2	51.2	57.3	19.0
Triplet	56.6	74.8	81.9	28.3	50.2	67.1	72.8	30.7
Ours	81.6	91.9	94.6	58.0	67.9	79.2	83.6	46.7

Table 3. Table 3. Comparison between 2048-dim and 4096-dim features. 2048-dim feature : 2048-dimension features from the Pool-5 layer. 4096-dim feature : 4096-dimension features from the 4096-dimension fully-connected layer.

Features	Duke to Market (%)				Market to Duke (%)				FlOPs (E9)	Params (million)
	rank-1	rank-5	rank-10	mAP	rank-1	rank-5	rank-10	mAP
2048-dim feature	81.6	91.9	94.6	58.0	67.9	79.2	83.6	46.7	2.698	25.557
4096-dim feature	81.3	91.5	94.1	58.5	67.5	78.5	82.1	46.0	2.715	33.958

Table 4. Table 4. Comparison with the state-of-the-art methods of domain adaptive re-ID on Market-1501 and DukeMTMC-reID. † † \dagger denotes semi-supervised methods. ‡ ‡ \ddagger denotes direct transfer methods for person re-ID.

Methods	DukeMTMC-reID to Market-1501 (%)				Market-1501 to DukeMTMC-reID (%)
	rank-1	rank-5	rank-10	mAP	rank-1	rank-5	rank-10	mAP
PTGAN(Wei et al., 2018)	38.6	-	66.1	-	27.4	-	50.7	-
PUL(Fan et al., 2018)	45.5	60.7	66.7	20.5	30.0	43.4	48.5	16.4
EUG $†$ (Wu et al., 2019)	49.8	66.4	72.7	22.5	45.2	59.2	63.4	24.5
CAMEL(Yu et al., 2017)	54.5	-	-	26.3	-	-	-	-
DGNet $‡$ (Zheng et al., 2019b)	56.1	72.2	78.1	26.8	42.6	58.6	64.6	24.3
MMFA(Lin et al., 2018)	56.7	75.0	81.8	27.4	45.3	59.8	66.3	24.7
SPGAN+LMP(Deng et al., 2018b)	57.7	75.8	82.4	26.7	46.4	62.3	68.0	26.2
TJ-AIDL(Wang et al., 2018b)	58.2	74.8	81.1	26.5	44.3	59.6	65.0	23.0
CamStyle(Zhong et al., 2019b)	58.8	78.2	84.3	27.4	48.4	62.5	68.9	25.1
HHL(Zhong et al., 2018)	62.2	78.8	84.0	31.4	46.9	61.0	66.7	27.2
ARN(Li et al., 2018)	70.3	80.4	86.3	39.4	60.2	73.9	79.5	33.4
ECN(Zhong et al., 2019a)	75.1	87.6	91.6	43.0	63.3	75.8	80.4	40.4
Ours	81.6	91.9	94.6	58.0	67.9	79.2	83.6	46.7

Table 5. Table 5. Comparison with the state-of-the-art methods of target-only re-ID on Market-1501 and DukeMTMC-reID.

Methods	Market-1501 (%)				DukeMTMC-reID (%)
	rank-1	rank-5	rank-10	mAP	rank-1	rank-5	rank-10	mAP
LOMO(Liao et al., 2015)	27.2	41.6	49.1	8.0	12.3	21.3	26.6	4.8
BOW(Zheng et al., 2015)	35.8	52.4	60.3	14.8	17.1	28.8	34.9	8.3
OIM(Xiao et al., 2017)	38.0	58.0	66.3	14.0	24.5	38.8	46.0	11.3
BUC(Lin et al., 2019)	66.2	79.6	84.5	38.3	47.4	62.6	68.4	27.5
Ours (target-only)	77.5	89.8	93.4	54.0	63.2	75.4	79.4	39.0

Table 6. Table 6. Comparison with the state-of-the-art methods on MSMT17. Src. denotes the source domain. * denotes that the results are reproduced by ourselves.

Methods	MSMT17 (%)
	Src.	rank-1	rank-5	rank-10	mAP
PTGAN (Wei et al., 2018)	Market	10.2	-	24.4	2.9
ECN (Zhong et al., 2019a)		25.3	36.3	42.1	8.5
Ours		25.5	37.3	42.6	9.2
PTGAN (Wei et al., 2018)	Duke	11.8	-	27.4	3.3
ECN (Zhong et al., 2019a)		30.2	41.5	46.8	10.2
Ours		32.3	44.4	50.1	11.7
BUC* (Lin et al., 2019)	N/A	11.5	18.6	22.3	3.4
Ours (target-only)		26.6	37.0	41.7	8.5

Equations

$$p(y_i^s|x_i^s)=\frac{\exp (g(\varphi (x_i^s;\mathbf{\theta });\mathbf{w})[y_i^s])}{{\sum }_{j=1}^J\exp (g(\varphi (x_i^s;\mathbf{\theta });\mathbf{w})[j])},$$

$$L_s=-\frac{1}{N_s}\sum _{i=1}^{N_s}\log p(y_i^s|x_i^s).$$

$$p(k|x_i^t)=\frac{\exp (\text{M}[k{]}^T\varphi (x_i^t;\mathbf{\theta })/\tau )}{{\sum }_{j=1}^{N_t}\exp (\text{M}[j{]}^T\varphi (x_i^t;\mathbf{\theta })/\tau )},$$

$$\text{M}[i]\leftarrow \mu \text{M}[i]+(1-\mu )\varphi (x_i^t;\mathbf{\theta }),$$

$$L_{\alpha }=-\frac{1}{N_t}\sum _{i=1}^{N_t}\log p(i|x_i^t).$$

$$L_{\beta 1}=-\frac{1}{N_t}\sum _{i=1}^{N_t}\sum _{j=1,j\ne i}^{N_t}{\mathbf{v}}_i^j\log p(j|x_i^t),$$

$$\begin{array}{r}\begin{array}{rl}{L}_{\gamma }=\sum _{i=1}^{N_t}\sum _{j=1}^{N_t}{\mathbf{v}}_i^j& \mathrm{d}(\text{M}[j],\varphi (x_i^t;\mathbf{\theta }))-\lambda \Vert {\mathbf{v}}_i{\Vert }_1,\\ & s.t.{\mathbf{v}}_i^j\in \{0,1\}\end{array}\end{array}$$

$$L_{\beta 2}=-\frac{1}{N_t}\sum _{i=1,\Vert {\mathbf{v}}_i{\Vert }_1\ge 2}^{N_t}\frac{1}{\Vert {\mathbf{v}}_i{\Vert }_1\log (\Vert {\mathbf{v}}_i{\Vert }_1)}\sum _{j=1,j\ne i}^{N_t}{\mathbf{v}}_i^j\log p(j|x_i^t).$$

$$\underset{\mathbf{v}}{\min }{L}_{\gamma }.$$

$$\underset{\mathbf{\theta },\mathbf{w}}{\min }\xi (L_{\alpha }+\delta L_{\beta 2})+(1-\xi )L_s,$$

Full text

person re-identification, unsupervised learning, domain adaptation, deep learning

††journalyear: 2020††doi: 10.1145/3369393††journal: TOMM††journalvolume: 16††journalnumber: 1††article: 3††publicationmonth: 2††ccs: Computing methodologies††ccs: Computing methodologies Visual content-based indexing and retrieval††ccs: Computing methodologies Image representations

1. Introduction

Person re-identification (re-ID) has become more and more popular because it plays an important role in security. Given an image of a person-of-interest from one camera, the goal of person re-ID is to find the person from other cameras. In recent years, benefiting from deep convolutional neural networks (CNNs) and large-scale datasets, person re-ID has made great progress. However, due to different cloth styles, camera viewpoints, scenes, etc., deep re-ID models often suffer from the problem of domain shift. Specifically, when we directly apply a pre-trained deep re-ID model to an unseen dataset, it often achieves considerably poor accuracy.

To address the domain-shift problem, one can collect a large amount of labeled data from the target domain for supervised training. However, it can be costly and time-consuming to collect a large-scale dataset. We can also collect a small amount of labeled data and a large amount of unlabeled data from the target domain for semi-supervised training (Wu et al., 2019; Liu et al., 2017b). However, when the number of target domains increases, this kind of methods are still impractical. Therefore, a promising solution is to use unlabeled data in target domains to train deep re-ID models, which is referred to as unsupervised person re-ID (Lin et al., 2019; Deng et al., 2018b; Wei et al., 2018; Fan et al., 2018; Wang et al., 2018b; Lin et al., 2018; Zhong et al., 2019b, a). This paper concerns unsupervised person re-ID.

There are two lines of unsupervised re-ID methods. The first line is to directly fine-tune a deep CNN model (usually pre-trained on ImagNet (Deng et al., 2009)) on the unlabeled target data (Li et al., 2014; Zheng et al., 2015, 2017c; Wei et al., 2018). In this paper, we call this line target-only re-ID (Lin et al., 2019). The second line is to exploit the labeled source data additionally. For example, PUL (Fan et al., 2018) first fine-tunes a deep CNN model on the labeled source dataset and then fine-tunes the model on the unlabeled target dataset in an unsupervised manner. ECN (Zhong et al., 2019a) fine-tunes the deep CNN model on the labeled source dataset and the unlabeled target dataset simultaneously. In the community, the second line is usually referred to as domain adaptive re-ID (Deng et al., 2018b; Wei et al., 2018; Fan et al., 2018; Wang et al., 2018b; Lin et al., 2018; Zhong et al., 2018, 2019b, 2019a). We propose an Adaptive Exploration (AE) method to improve the deep re-ID model on the target domain. We mainly discuss AE with the domain adaptive re-ID protocol. However, it can also achieve comparable accuracy on the target-only re-ID.

To learn discriminative features in the target domain, AE is designed to maximize distances between all person images and minimize distances between similar person images in the feature space. We leverage a non-parametric classifier, equipped with a feature memory, to achieve this objective. The feature memory stores the features of all target person images. After each iteration, the memory is updated according to the newly learned person image features. The non-parametric classifier classifies person images according to its feature memory. Specifically, to maximize distances between all person images, the classifier treats each person image as an individual class. Given the feature and the index of a person image, by a softmax, the classifier tries to minimize the distance between the newly learned feature and its old feature in the memory, while maximizing the distances between the newly learned feature and other person image features in the memory. To minimize distances between a group of similar person images, the classifier treats them as the same class.

However, it is challenging to select reliable similar samples in an unsupervised manner. Especially in the early training stage, it is impossible to find a large number of reliable similar person images that are located near to each other in the feature space. To alleviate this problem, the AE method adaptively selects a small amount of reliable similar samples at the beginning of the training, according to a similarity threshold. When the model becomes stronger, more person images are adaptively selected as reliable samples, which in return benefits the training for the deep re-ID model. In this way, the model is progressively improved. With this adaptive selection, one problem is that the method may result in an unbalanced neighborhood distribution. Once a person image has more neighborhoods than others, the sum of its losses will be larger, which force other images to move to it quickly. In consequence, images with more neighborhoods will make more images become their neighborhoods, which inevitably contains unreliable and noisy person images. To alleviate this problem, we integrate a balance strategy into the method. A penalty is employed into the loss function. The penalty automatically changes the weight on the loss to make training balanced.

The contributions of this article are twofold:

• •

We propose an AE method for unsupervised person re-ID. The method maximizes distances of all target images and minimizes distances of similar target images. With a similarity threshold and a balance term, the method can adaptively find reliable similar target images.

• •

Extensive experiments on three large-scale datasets demonstrate the effectiveness of AE. Our method significantly improves the unsupervised person re-ID accuracy, including both target-only re-ID and domain adaptive re-ID.

2. Related Work

In this section, we briefly recount the previous work of person re-ID, including supervised person re-ID and unsupervised person re-ID. In this paper, we separate unsupervised person re-ID into target-only re-ID and domain adaptive re-ID.

2.1. Supervised Person Re-identification

In recent years, deep learning has promoted great development in many computer vision tasks (Song et al., 2018c, a). Compared with hand-crafted feature methods (Farenzena et al., 2010; Zhao et al., 2013; Köstinger et al., 2012; Zheng et al., 2013; Liao et al., 2015; Zheng et al., 2015; Wang et al., 2018a), deep learning methods have dominated the person re-ID community. Based on deep networks, supervised person re-ID has been extensively studied from general problems to specific problems.

For general CNN model, the siamese model (Yi et al., 2014; Varior et al., 2016b, a; Li et al., 2014; Cheng et al., 2016; Shi et al., 2016) and the classification model (Zheng et al., 2017b; Zheng et al., 2016) are studied. Among the work of siamese model exploration, Yi et al. (Yi et al., 2014) and Li et al. (Li et al., 2014) firstly employed a siamese network for person re-ID and utilized part information in model training. For the classification model, Zheng et al. (Zheng et al., 2017b) used a conventional fine-tuning approach, the ID-discriminative embedding (IDE). Recently, Quan et al. (Quan et al., 2019) used Neural Architecture Search (NAS) to search an efficient and effective CNN architecture for person re-ID. In addition to the deep network, ranking loss (Chen et al., 2016; Cheng et al., 2016; Chen et al., 2017; Hermans et al., 2017) and classification loss (Li et al., 2014; Varior et al., 2016a; Wang et al., 2016) are also explored. Besides, to alleviate the over-fitting problem, several data augmentation methods have been proposed (Zhong et al., 2017; Zheng et al., 2017c; Zhong et al., 2019b). Among these data augmentation methods, Zhong et al. (Zhong et al., 2019b) changed camera styles for images as data augmentation to enhance the robustness of the model to camera variance.

Besides general problems, some specific problems of person re-ID are studied recently, such as pose variance problem (Su et al., 2017), viewpoint variance problem (Sun and Zheng, 2019), background variance problem (Tian et al., 2018) and occlusion problem (Miao et al., 2019). To alleviate the pose variance problem, several methods (Farenzena et al., 2010; Zheng et al., 2019a; Cho and Yoon, 2016; Su et al., 2017; Sarfraz et al., 2018) are proposed to learn pose invariant representations. Su et al. (Su et al., 2017) employed a pose-driven deep convolutional model to leverage the human part cues for person re-ID. For viewpoint variance problem, some work (Wu et al., 2015; Karanam et al., 2015; Zheng et al., 2017a; Sun and Zheng, 2019) focus on it. Among them, Sun et al. (Sun and Zheng, 2019) quantitatively analyzed and revealed the impacts of different viewpoints. To alleviate the background variance problem, several methods (Song et al., 2018b; Tian et al., 2018; Xiao et al., 2017) is designed to enhance the robustness of the re-ID model to background noises. Tian et al. (Tian et al., 2018) proposed a person-region guided pooling network and random background augmentation to alleviate background bias.

To alleviate domain bias, in this work, we focus on unsupervised person re-ID.

2.2. Target-only Person Re-identification

Target-only person re-ID (Xiao et al., 2017; Lin et al., 2019) only leverages unlabeled target data during training. Although without the deep CNN, two hand-crafted feature methods (Liao et al., 2015; Zheng et al., 2015) are also classified by us as target-only person re-ID methods. It is because the hand-crafted feature can be directly used for the person re-ID without any training data. In addition to hand-crafted feature methods, Xiao et al. (Xiao et al., 2017) introduced the feature memory with online instance matching (OIM) loss. These can be used in target-only person re-ID. Lin et al. (Lin et al., 2019) introduced bottom-up clustering (BUC) and achieved competitive accuracy with only unlabeled target data.

2.3. Domain Adaptive Person Re-identification

Domain adaptive person re-ID (Deng et al., 2018b; Wei et al., 2018; Fan et al., 2018; Wang et al., 2018b; Lin et al., 2018; Zhong et al., 2018, 2019b, 2019a; Yu et al., 2017; Li et al., 2018) exploits both labeled source data and unlabeled target data. Among them, Yu et al. (Yu et al., 2017) proposed clustering-based asymmetric metric learning called CAMEL. Fan et al. (Fan et al., 2018) introduced a progressive algorithm with clustering and fine-tuning for re-ID model training. Deng et al. (Deng et al., 2018b) and Wei et al. (Wei et al., 2018) transferred source data to the target domain with CycleGAN (Zhu et al., 2017) and employed the generated data to train the re-ID model. Wang et al. (Wang et al., 2018b) and Lin et al. (Lin et al., 2018) incorporated attribute information to enhance the scalability and usability of the re-ID model. Zhong et al. (Zhong et al., 2019a) considered the intra-domain variation for the target domain and kept three underlying invariances during training, that is, exemplar-invariance, camera-invariance, and neighborhood-invariance. When keeping neighborhood-invariance, they fixed the number of neighborhoods for each target image. However, this might be not reasonable during model training. For example, in the early stage of training, only a few reliable neighborhoods can be found near each target image in the feature space. In this case, fewer neighborhoods should be selected. Besides, in the later stage of training, more reliable neighborhoods move close to the target images in the feature space. In this case, more neighborhoods can be selected. Therefore, in this work, an adaptive selection of neighborhoods is employed. However, adaptive selection leads to different numbers of neighborhoods for different target images which may lead to an unbalanced training. Thus, a balanced term is employed in our method. With the adaptive selection and the balance term, the AE method manages to achieve competitive accuracy on both domain adaptive person re-ID and target-only person re-ID.

3. Adaptive Exploration Method

In this section, we introduce the adaptive selection (AE) method in detail. The framework is shown in Figure 1. Source images and target images are input into a convolutional neural network (CNN) simultaneously. The difference is that, with the labeled source images, supervised learning is used to train the model to obtain a basic re-ID capability. With the unlabeled target images, as shown in Figure 1, the AE method is proposed to help the model to generalize well in the target domain. Using a non-parametric classifier equipped with a feature memory, our method simultaneously maximizes distances of all target images and minimizes distances of similar target images in the feature space. In the end, our model can learn discriminative features for person images in the target domain.

In Section 3.1, we introduce supervised learning with parametric classifier in the source domain. In Section 3.2, we introduce the non-parametric classifier, which is equipped with a feature memory and used for the target domain. In Section 3.3, we introduce the formulation of learning with adaptive selection, including maximizing distances between all images, minimizing distances between similar images, and adaptive selection of similar images. In Section 3.4, we introduce a balance strategy, which aims to make the number of neighborhoods balanced and reasonable. In Section 3.5, we introduce the optimization procedure of our method.

3.1. Parametric Classifier for Source Domain

In the source domain, supervised learning with parametric classifier is used. Suppose the source domain contains $N_{s}$ labeled images $\{(x^{s}_{1},y^{s}_{1}),(x^{s}_{2},y^{s}_{2}),...,(x^{s}_{N_{s}},y^{s}_{N_{s}})\}$, where $s$ denotes the source domain. The probability that the $x^{s}_{i}$ image belongs to the $y^{s}_{i}$ identity is defined as follows,

[TABLE]

where $\phi(\cdot;\bm{\theta})$ denotes our model, extracting features for every image, $\bm{\theta}$ denotes weights of the deep re-ID model, $g(\cdot;\mathbf{w})$ denotes the parametric classifier, $\mathbf{w}$ denotes the weights of the classifier, $g(\phi(x_{i}^{s};\bm{\theta});\mathbf{w})[j]$ denotes the $j$-th logit of the output from the classifier given the feature $\phi(x_{i}^{s};\bm{\theta})$ and $J$ denotes the number of person identities of the source dataset. The training for the source domain is to minimize the following loss,

[TABLE]

3.2. Non-parametric Classifier with Feature Memory

To learn discriminative features for the target domain, our method tries to maximize feature distances between all person images and minimize feature distances between similar person images. To optimize distances, one can use contrastive loss (Varior et al., 2016a) or triplet loss (Hermans et al., 2017; Liu et al., 2017a). However, these losses become less effective when datasets become large. Therefore, we try to optimize distances under a classification framework. For example, to minimize feature distances between similar person images, we can treat them as the same class. To maximize feature distances between all target person images, we can treat each image as an individual class.

However, treating each image as an individual class may make the parametric classifier difficult to converge. To alleviate this problem, motivated by (Wu et al., 2018b; Wu et al., 2018a; Xiao et al., 2017; Zhong et al., 2019a; Yang et al., 2019), we exploit a non-parametric classifier, which is equipped with a memory M, to classify target images.

Suppose the target domain contains $N_{t}$ unlabeled images $\{x^{t}_{1},x^{t}_{2},...,x^{t}_{N_{t}}\}$, where $t$ denotes the target domain. After feature extraction, each image is embedded as a $D$-dimensional vector. The feature memory $\textbf{M}\in\mathbb{R}^{D\times N_{t}}$ stores all target image features and is updated after each training iteration.

Based on the memory M, given an image, the non-parametric classifier aims to produce the probability of the image being the same class as other images. For example, the probability that the $x_{i}^{t}$ image is same as the $k$-th image is defined as follows,

[TABLE]

where $\textbf{M}[k]$ denotes the $k$-th column of the feature memory M, representing the feature of the $k$-th image, and the hyper-parameter $\tau$ denotes the temperature fact of the softmax function. A higher temperature $\tau$ leads to a softer probability distribution. After each iteration, M is updated as follows,

[TABLE]

where the hyper-parameter $\mu$ is the update rate of M.

Instead of fixing $\mu$ to a constant value, we increase it linearly as the number of epochs increasing. Since M is not reliable enough at the beginning of training, a smaller $\mu$ is needed to accelerate the update of M. By rapidly updated with newly learned representations, M can memorize discriminative features quickly. As M becomes discriminative gradually, M is required to be more stable. Therefore, in this time, a larger $\mu$ is used to slow down the update.

3.3. Learning with Adaptive Selection

In this subsection, we introduce the learning with adaptive selection for the target domain, including maximizing distances between all target images, minimizing distances between similar target images, and adaptive selection of similar images.

We firstly introduce maximizing distances between all target images. To achieve this objective, we assume each target image as an individual class. Specifically, the index of each target image is treated as its pseudo label. We try to maximize the distances between target images by minimizing the following loss,

[TABLE]

When applying this loss to a specific image, the image is encouraged to move far away from other images (Figure 1). When applying this loss to all images, they are encouraged to move far away from each other. The feature distances between all target images are thus maximized.

To minimize distances between similar target images, we first use a similarity threshold to adaptively select reliable neighborhoods for each target image. Only neighborhoods whose distances to the given target image are smaller than the threshold are selected as reliable neighborhoods. Secondly, we assume that the target image and its reliable neighborhoods share the same person identity, i.e., treating the image and its reliable neighborhoods as the same class. By this operation, each target image is forced by the non-parametric classifier to move closer to its neighborhoods (Figure 1), which makes similar target images stay closer. The loss is defined as follows,

[TABLE]

where $\mathbf{v}_{i}\in\{0,1\}^{N_{t}}$ denotes the selection indication vector of $i$-th target image. Specifically, $\mathbf{v}_{i}^{j}=1$ indicates that the $j$-th image is selected as a neighborhood of the $i$-th image. When $\mathbf{v}_{i}^{j}=0$, the $j$-th image will not be used when forcing the $i$-th image move closer to its neighborhoods.

For neighborhoods selection, we are inspired by active learning (Yan et al., 2016; Deng et al., 2019, 2018a) and Self-Pace Learning (SPL) (Kumar et al., 2010), which has been widely used in weakly-supervised learning (Fan et al., 2017), semi-supervised learning (Fan et al., 2018; Ma et al., 2017) and unsupervised learning (Fan et al., 2018). The basic idea of SPL is to incorporate from easy samples to hard samples during training. Samples with small losses are considered as reliable samples. In this paper, reliable or easy neighborhoods are needed to be selected for minimizing feature distances between similar person images. We select reliable neighborhoods according to their distances or similarities to the given image. Specifically, when an image is close enough to the given image in the feature space, it is selected as a reliable neighborhood to the given image. We formulate this selection as minimizing the following loss,

[TABLE]

where $\lambda$ is the similarity threshold, $\mathrm{d}(\cdot)$ denotes a distance function and $\|\mathbf{v}_{i}\|_{1}$ denotes the number of selected neighborhoods for the $i$-th image. By minimizing $L_{\gamma}$, two images will be treated as neighborhoods if their distance is less than $\lambda$.

3.4. Learning with Balance

A problem caused by Eq. (6) is that the number of neighborhoods for an image can change dramatically. When an image has a large number of neighborhoods, the sum of the losses between it and its neighborhoods can be considerably large. When an image has a small number of neighborhoods, the sum of the losses between it and its neighborhoods can be very small. As a consequence, as shown in Figure 2(a), it is easier for a large group to attract other images than a small group, which makes most groups have only a few data instances while a minority of groups have a large number of data instances. This unbalanced learning may result in a poor accuracy for re-ID.

To address this issue, we integrate a balance term into Eq. (6) to make the adaptive selection balanced,

[TABLE]

When $\|\mathbf{v}_{i}{\|}_{1}=1$, the image does not have any other neighborhoods expect itself. Therefore, we don’t use it as a training sample and only consider the cases when $\|\mathbf{v}_{i}{\|}_{1}\geq 2$.

When an image has a large number of neighborhoods, the penalty $1/(\|\mathbf{v}_{i}\|_{1}\mathrm{log}(\|\mathbf{v}_{i}\|_{1}))$ decreases the losses between the image and its neighborhoods heavily. Otherwise, the losses are decreased slightly (Figure 2(b)). By this balance strategy, no matter whether it’s a large group or a small group, the group attracts images in a relatively similar degree. Therefore, they will have a similar number of neighborhoods. During training, the number of neighborhoods thus becomes balanced and reasonable.

3.5. Optimization Procedure

During training, we alternately optimize the involved parameters, i.e., v and ($\bm{\theta},\mathbf{w}$).

1) Optimize $\mathbf{v}$ when $(\bm{\theta},\mathbf{w})$ is fixed. The goal of this step is to adaptively select reliable neighborhoods for minimizing distances between them in the feature space, which is achieved by minimizing

[TABLE]

Specially, if the distance between two images is below the threshold $\lambda$, they will be chosen as neighborhoods for each other. The details are provided in Algorithm 1.

2) Optimize $(\bm{\theta},\mathbf{w})$ when $\mathbf{v}$ is fixed. This step utilizes source data and target data to train the re-ID model by minimizing

[TABLE]

where hyper-parameter $\delta$ and $\xi$ aim to control the importance of these losses.

3) Update the feature memory $\mathbf{M}$. In this step, $\mathbf{M}$ for the non-parametric classifier is updated by Eq. (4).

With the optimization procedure, our model manages to recognize people in the target domain. Note that, with the target-only re-ID protocol, the loss from source domain is not used and thus the $\xi$ in Eq. (10) is set to 1.

4. Experiment

In this section, we evaluate the proposed method on three large-scale re-ID datasets. Besides Market-1501 and DukeMTMC-reID, which are widely used by most existing methods, we also report the accuracy on the MSMT17 dataset.

4.1. Datasets and Settings

Market-1501 (Zheng et al., 2015) contains 32,688 images with 1,501 identities. They are captured by 6 cameras on campus. The dataset is split into three parts: 12,936 images with 751 identities for training, 19,732 images with 750 identities for the gallery, and another 3,368 images with the same 750 gallery identities for query.

DukeMTMC-reID (Zheng et al., 2017c) contains 36,411 images with 1,812 identities, which are collected from 8 cameras. Following Market-1501, the dataset is split into three parts: 16,522 images with 702 identities for training, 17,661 images with 1,110 identities in the gallery, and another 2,228 images with the same 702 identities as the gallery for query.

MSMT17 (Wei et al., 2018) contains 126,441 images with 4,101 identities, which are captured by 15 cameras. Similar to Market-1501 and DukeMTMC-reID, the dataset is split into three parts: 32,621 images with 1,041 identities for training, 82,161 images with 3,060 identities in the gallery and another 11,659 images with the same 3,060 gallery identities for query.

We reported the rank1, rank5, rank10 and mean average precision (mAP) for evaluation on the three datasets. All experiments used single-query.

4.2. Implementation Details

We used ResNet-50 (He et al., 2016) as the CNN backbone to extract features. The ResNet-50 was pre-trained on ImageNet (Deng et al., 2009). After the Pool-5 layer of the ResNet-50, we added a 4,096-dimension fully-connected layer followed by batch normalization (Ioffe and Szegedy, 2015), ReLU (Nair and Hinton, 2010) and Dropout (Srivastava et al., 2014). Therefore, the length of re-ID features for training is 4,096. Note that, during testing, following most existing methods, we used the 2,048-dimension feature from the Pool-5 layer. During evaluation, the 2048-dimension feature is able to achieve similar accuracy as the 4096-dimension feature, but consumes fewer flops and parameters . During training, we fixed the first two layers of the ResNet-50. After feature extraction, two different classifiers were used to classify person images. For source images, we adopted a general parametric classifier for supervised learning. For target images, we adopted a non-parametric classifier with a feature memory M.

We used random flip, crop and erasing (Zhong et al., 2017) as data augmentation for both target images and source images. Images were resized to 256 $\times$ 128. For each iteration, we randomly chose 128 images from the target domain and 128 images from the source domain. We also leveraged the CamStyle (Zhong et al., 2019b) method to reduce camera variance in the target domain.

We used Stochastic Gradient Descent (SGD) to train the model. Weight decay and momentum were set to $5\times 10^{-4}$ and 0.9, respectively. Learning rate was set to 0.01 for ResNet-50 base layers and 0.1 for other layers in the first 40 epochs. The learning rate was then divided by 10 in the next 20 epochs. For the target domain, we started minimizing distances of similar images after 5 epochs. During training, we used cosine to measure similarity. The 4,096-dimension features are $l_{2}$-normalized before stored into the memory M. Two images will be selected as neighborhoods when their cosine similarity is larger than $\lambda$.

For hyper-parameters, we set the similarity threshold $\lambda$ and the temperature $\tau$ to 0.55 and 0.05, respectively. By default, with the number of epochs increasing, the memory update rate $\mu$ was linearly increased from 0 to 0.4 for Market-1501 and from 0 to 0.5 for DukeMTMC-reID. The $\delta$ and $\xi$ was set to 3.5 and 0.6, respectively.

4.3. Ablation Study

4.3.1. Exploration for Update Rate $\mu$

To investigate the effect of $\mu$, different varying rates of $\mu$ were evaluated. We linearly increased the varying $\mu$ from 0 to 0.1, …, 0.9, respectively. Meanwhile, the constant $\mu$ was also used to be compared with the varying $\mu$. The $\mu$ was fixed to 0.1, …, 0.9, respectively. The results are shown in Figure 3(a) and Figure 3(b).

First, the varying $\mu$ is superior to the constant $\mu$. The accuracy curves of the varying $\mu$ are often above those of the constant $\mu$.

Secondly, for varying $\mu$, different varying rates lead to different accuracy. In Figure 3(a) and Figure 3(b), as the varying rates increase, accuracy curves increase at first and then decrease. In Figure 3(a), when tested on Market-1501, the peak is achieved when $\mu$ is increased from 0 to 0.4. In Figure 3(b), when tested on DukeMTMC-reID, the peak is achieved when $\mu$ is increased from 0 to 0.5.

4.3.2. Learning with Adaptive Exploration

To investigate the effect of learning with adaptive exploration, we adopted different values of similarity threshold $\lambda$. The accuracy is reported in Figure 3(c). The average number of selected neighborhoods in the last epoch is shown in Figure 3(d). The comparisons between adaptive selection and top-$k$ are shown in Figure 3(e) and Figure 3(f).

In Figure 3(c), different values of $\lambda$ result in different accuracy. With $\lambda$ increasing, accuracy curves (rank-1 accuracy and mAP) increase first and then decrease. The highest accuracy is achieved when $\lambda$ is equal to 0.55. The best $\lambda$s for different datasets are similar. Specifically, high accuracy is achieved when $\lambda=0.55$ on both Market-1501 and DukeMTMC-reID.

In Figure 3(d), when $\lambda=0.55$, the average number of selected neighborhoods is close to the ground truth. For example, the Market-1501 training set contains 12,936 images with 751 person identities. The average number of images per person is 17.2. When $\lambda=0.55$, our model selects 18.3 neighborhoods on average in the last epoch, which is close to the ground truth 17.2.

In Figure 3(e) and Figure 3(f), top-$k$ is always under adaptive selection. For example, when transferring DukeMTMC-reID to Market-1501 in Figure 3(e), there exists a large margin between mAP curves of top-$k$ and adaptive selection. This demonstrates that the adaptive selection can significantly improve the model.

4.3.3. Learning with Balance

To investigate the effect of learning with balance, we compared models with balance and models without balance in Table 1. Without balance, the model achieves higher accuracy when we set $\delta$ to 0.2 and increased $\mu$ from 0 to 0.4.

Experimental results suggest that the usage of learning with balance significantly improves accuracy. For example, when transferring DukeMTMC-reID to Market-1501 with CamStyle augmentation, we observe +15.0% in rank-1 accuracy and +22.9% in mAP with the balance strategy.

4.3.4. Comparison between Transfer Learning and Target-only Learning

In this paper, we considered both domain adaptive re-ID and target-only re-ID of our method. We reported their results in Table 1.

Firstly, the source-only model fails to produce good results in the target domain. For example, when directly transferring DukeMTMC-reID to Market-1501, the source-only model only achieves 46.4% in rank-1 accuracy and 20% in mAP. When directly transferring Market-1501 to DukeMTMC-reID, the source-only model only achieves 30.3% in rank-1 accuracy and 15.9% in mAP. This demonstrates the notorious domain bias problem.

Secondly, our model significantly increases accuracy with transfer learning. For example, when transferring DukeMTMC-reID to Market-1501, our model achieves 81.6% in rank-1 accuracy and 58.0% in mAP. Compared with the source-only baseline model, our model increases the accuracy by 35.2% in rank-1 accuracy and 38.0% in mAP. When transferring Market-1501 to DukeMTMC-reID, our model achieves 67.9% in rank-1 accuracy and 46.7% in mAP. Compared with the source-only baseline model, our model increases the accuracy by 37.6% in rank-1 accuracy and 20.8% in mAP.

At last, the target-only learning of our method still achieves competitive accuracy. For example, on Market-1501, our model with the target-only re-ID protocol achieves 77.5% in rank-1 accuracy and 54.0% in mAP. Compared with the transfer learning, the target-only learning decreases rank-1 accuracy and mAP by 4.1% and 4.0%, respectively.

4.3.5. Hyper-parameter Analysis

In Figure 4, we analyzed the impacts of three hyper-parameters, i.e., $\tau$, $\xi$ and $\delta$. We varied one hyper-parameter and fixed others to train re-ID models.

In Figure 4(a), we changed $\tau$ from 0.01 to 1.0. As shown in the figure, the higher accuracy is achieved when $\tau$ is equal to 0.05.

In Figure 4(b), we changed $\xi$ from 0.0 to 1.0. When $\xi$ is set to 0.0 or 1.0, the accuracy decreases. This demonstrates only using source data ($\xi$ = 0.0) or only using target data ($\xi$ = 1.0) achieve lower accuracy than using them together. Besides, when $\xi\in$ [0.3, 0.9], the accuracy is stable.

In Figure 4(c), we changed $\delta$ from 0.0 to 10.0. When $\delta\in$ [3.0, 4.0], the higher accuracy is achieved.

4.3.6. Comparison between Triplet and Contrastive Loss

In the target domain, the re-ID model is improved by maximizing feature distances between all images and minimizing feature distances between similar images. Besides the classification framework in AE, triplet loss and contrastive loss can also be used to achieve this objective. Specifically, for triplet loss, each target image and its neighborhoods are treated as positive pairs while others are viewed as negative samples. Similarly, for contrastive loss, feature distances between target images and their neighborhoods are minimized while feature distances between target images and the negative samples are maximized. Compared with AE, as shown in Table 2, triplet loss and contrastive loss fail to achieve competitive accuracy, demonstrating that our method is more effective than triplet loss and contrastive loss.

4.3.7. 2048-dim Features vs. 4096-dim Features

During test, we utilized 2048-dimension features from the Pool-5 layer to retrieve images. In addition, we also used 4096-dimension features from the 4096-dimension fully-connected layer. As shown in Table 3, the two features achieve similar accuracy. However, compared with 2048-dimension features, 4096-dimension features use one more layer for computation, i.e., the 4096-dimension fully-connected layer. Therefore, as shown in Table 3, more flops and parameters are used for 4096-dimension features.

4.4. Comparison with State-of-the-art Methods

We compared our method with the state-of-the-art unsupervised person re-ID methods on Market-1501, DukeMTMC-reID, and MSMT17. Experimental results are shown in Table 4, Table 5, and Table 6, respectively. Table 4 and Table 5 report the results of the domain adaptive re-ID methods and the target-only re-ID methods, respectively, on Market-1501 and DukeMTMC-reID. As for MSMT17, the results are shown in Table 6.

In Table 4, our method is compared with ten domain adaptive methods of person re-ID. Compared with the state-of-the-art domain adaptive re-ID method, i.e., ECN (Zhong et al., 2019a), our method increases rank-1 accuracy by 6.5% and 4.6% when tested on Market-1501 and DukeMTMC-reID, respectively.

In Table 5, we compared our method with four target-only re-ID methods (LOMO(Liao et al., 2015), BOW(Zheng et al., 2015), OIM(Xiao et al., 2017) and BUC (Lin et al., 2019)). When tested on Market-1501, LOMO and BOW achieve 27.2% and 35.8% in rank-1 accuracy, respectively. Compared with these methods, the target-only AE method yields the highest accuracy on Market-1501 and DukeMTMC-reID. Specifically, on Market-1501, AE achieves 77.5% in rank-1 and 54.0% in mAP. On DukeMTMC-reID, AE achieves 63.2% in rank-1 and 39.0% in mAP. Compared with the state-of-the-art target-only re-ID method, i.e., BUC (Lin et al., 2019), AE increases rank-1 accuracy by 11.3% when tested on Market-1501 and 15.8% when tested on DukeMTMC-reID.

In Table 6, we compared our method with two domain adaptive re-ID methods (PTGAN (Wei et al., 2018) and ECN (Zhong et al., 2019a)) and one target-only method (BUC (Lin et al., 2019)) on MSMT17.

The target-only AE method achieves competitive results on MSMT17 in Table 6. Specifically, the target-only AE method achieves 26.6% in rank-1 accuracy and 8.5% in mAP on MSMT17. Compared with the target-only person re-ID method BUC (Lin et al., 2019), the target-only AE method increases rank-1 accuracy and mAP by 15.1% and 5.1%, respectively.

4.5. Visualization of Feature Space

4.5.1. Effectiveness of Learning with Balance

To additionally investigate the effectiveness of learning with balance, we use PCA to visualize neighborhoods selected in the last epoch (60 epoch) by two images on DukeMTMC-reID. The results are shown in Figure 5.

In Figure 5(a), without the balance term, one image selects too many neighborhoods while the other one only chooses a few neighborhoods. Meanwhile, the identities of some neighborhoods are different from those of the two images. In Figure 5(b), when learning with balance, the two images enable to select a similar number of neighborhoods. Also, these neighborhoods share the same identities with the two images. This indicates that learning with balance manages to help our model classify people accurately.

4.5.2. Effectiveness of the AE Method

To additionally investigate the effectiveness of our method, we use t-SNE (Maaten and Hinton, 2008) to visualize feature distributions shown in Figure 6. Specifically, part of gallery images on Market-1501 (1926 images with 100 identities) are extracted into features and, then the features are projected into a 2-dimension map by t-SNE. Note that, each point in the map represents one image and points with the same color indicate the same person images.

In Figure 6(d), same color points often stay together and are far away from other color points. This demonstrates our model can extract discriminative features. Caused by the lack of labels, our model inevitably classifies two similar persons as one identity. Therefore, in Figure 6(d), there exist two different color points being together.

We also visualize feature distributions from three other methods, that is, source-only, BUC (Lin et al., 2019) and ECN (Zhong et al., 2019a) shown in Figure 6(a), Figure 6(b) and Figure 6(c), respectively. Source-only indicates the baseline re-ID model fine-tuned on only source data. BUC (Lin et al., 2019) and ECN (Zhong et al., 2019a) are the current best methods of target-only re-ID and domain adaptive re-ID, respectively. Compared with them, our method results in better feature distributions in Figure 6(d). Specifically, same color points stay closer and fewer different color points stay together by mistake. This demonstrates the superiority of our method.

5. Conclusion

In this paper, we propose the adaptive exploration (AE) method for unsupervised person re-ID. The AE method explores the unlabeled target domain by considering the feature distances between target images. By a non-parametric classifier with a feature memory, AE maximizes distances of all target images and minimizes distances of similar target images. Meanwhile, we propose to employ a similarity threshold to select reliable similar images. However, with adaptive selection, some images select too many neighborhoods while others only have a few neighborhoods. To alleviate the unbalanced problem, we integrate a balance term into the objective loss to prevent images, which have too many neighborhoods, from attracting other images. As a result, each image tends to select a balanced and reasonable number of neighborhoods. With the adaptive selection and the balance term, the AE method achieves competitive accuracy on both target-only and domain adaptive re-ID.