Group Re-Identification with Multi-grained Matching and Integration

TL;DR

This paper introduces a novel multi-grained representation and dynamic matching approach for robust group re-identification across camera views, addressing challenges like pose, layout, and membership variations.

Contribution

It proposes a new multi-grained group representation and a multi-order matching method with iterative weight updates for improved group Re-ID accuracy.

Findings

Effective on complex multi-camera datasets

Outperforms existing group Re-ID methods

Demonstrates robustness to layout and membership changes

Abstract

The task of re-identifying groups of people underdifferent camera views is an important yet less-studied problem.Group re-identification (Re-ID) is a very challenging task sinceit is not only adversely affected by common issues in traditionalsingle object Re-ID problems such as viewpoint and human posevariations, but it also suffers from changes in group layout andgroup membership. In this paper, we propose a novel conceptof group granularity by characterizing a group image by multi-grained objects: individual persons and sub-groups of two andthree people within a group. To achieve robust group Re-ID,we first introduce multi-grained representations which can beextracted via the development of two separate schemes, i.e. onewith hand-crafted descriptors and another with deep neuralnetworks. The proposed representation seeks to characterize bothappearance and spatial relations of multi-grained objects, and isfurther equipped with importance weights which capture varia-tions in intra-group dynamics. Optimal group-wise matching isfacilitated by a multi-order matching process which in turn,dynamically updates the importance weights in iterative fashion.We evaluated on three multi-camera group datasets containingcomplex scenarios and large dynamics, with experimental resultsdemonstrating the effectiveness of our approach. The published dataset can be found in \url{http://min.sjtu.edu.cn/lwydemo/GroupReID.html}

Tables

Table 1. Table I: Statistical Analysis of Datasets

Datasets	i-LID MCTS	Road Group	DukeMTMC Group
Average Person	2.313	3.812	3.392
Member Change	0.187	0.451	0.832
Dispersity	0.317	0.376	0.407
Occlusion Pairs	1.021	1.775	2.001

Table 2. Table II: Ablation study results of representations of various granularities on the Road Group dataset.

Rank (hand-crafted)	1	5	10	15	20
Global	15.8	31.6	43.0	48.6	54.8
Fine	62.0	82.2	89.6	95.1	96.5
Fine+Medium	66.7	87.2	93.3	96.0	96.8
Fine+Medium+Coarse	71.1	89.4	94.1	97.0	97.3
Proposed-equal weights	55.8	78.0	88.1	92.1	93.6
Proposed-no spatial	69.6	88.6	94.0	96.2	96.5
Proposed-auto	72.3	90.6	94.1	97.1	97.5
Proposed-GT	76.0	91.8	95.3	97.2	98.0
Rank (deep convolution)	1	5	10	15	20
Global	32.1	67.9	77.8	84.0	86.4
Fine	69.1	88.9	92.3	93.8	95.1
Fine+Medium	69.1	90.1	95.1	96.3	96.3
Fine+Medium+Coarse	72.8	93.8	95.1	96.3	96.8
Proposed-equal weights	72.4	90.1	92.6	96.3	97.5
Proposed-no spatial	70.4	90.1	91.3	92.6	96.3
Proposed-auto	80.2	93.8	96.3	97.5	97.5
Proposed-GT	82.4	95.1	96.3	97.5	98.0

Table 3. Table III: CMC results for group Re-ID on different datasets based on various feature combinations

	i-LIDS MCTS					DukeMTMC Group					Road Group
	1	5	10	15	20	1	5	10	15	20	1	5	10	15	20
Hand-craft	37.9	64.5	79.4	91.5	93.8	47.4	68.1	77.3	83.6	84.7	72.3	90.6	94.1	97.1	97.5
Global-conv	31.3	56.9	73.1	85.6	91.2	42.1	67.8	79.0	84.6	86.2	75.3	93.8	96.3	96.3	97.5
Appearance-conv	35.6	66.2	80.6	87.9	95.0	44.9	73.1	82.7	89.9	93.3	76.8	92.3	95.1	97.5	97.5
Full-conv	38.8	65.7	82.5	93.8	98.8	48.4	75.2	89.9	93.3	94.4	80.2	93.8	96.3	97.5	97.5

Table 4. Table IV: Ablation study results of various matching order and criterion on the Road Group dataset

Rank (hand-crafted)	1	5	10	15	20
Single	62.0	82.2	89.6	95.1	96.5
No inter	70.1	88.8	94.1	96.3	97.5
No dis	65.8	88.8	93.8	96.3	96.3
Hyp-E [39]	55.1	77.8	87.6	88.9	95.1
Proposed-auto	72.3	90.6	94.1	97.1	97.5
Rank (deep convolution)	1	5	10	15	20
Single	69.1	88.9	92.3	93.8	95.1
No inter	74.1	92.6	96.3	97.5	97.5
No dis	66.7	90.1	95.8	96.3	97.5
Hyp-E [39]	62.9	77.8	88.9	95.1	96.3
Proposed-auto	80.2	93.8	96.3	97.5	97.5

Table 5. Table V: Rank-1 Results (R1) of different matching schemes for single person Re-ID on the Road Group Dataset

	Methods	Rank-1 Score
Without Group	TriNet [12]	37.2
	AlignReID [26]	32.8
	Single-match (hand-crafted)	26.5
	Single-match (deep conv)	21.0
With Group	Intra-group (hand-crafted)	67.1
	Intra-group (deep conv)	70.3
	Proposed-auto (hand-crafted)	71.4
	Proposed-auto (deep conv)	73.5

Table 6. Table VI: CMC results for group Re-ID on the three datasets

	i-LIDS MCTS					DukeMTMC Group					Road Group
	1	5	10	15	20	1	5	10	15	20	1	5	10	15	20
Saliency [6]	26.1	48.5	67.5	80.3	89.9	13.9	33.3	51.5	59.8	66.3	48.6	73.6	82.2	86.2	90.1
Mirror+KMFA [15]	28.3	58.4	69.8	80.5	90.6	11.0	31.5	49.7	62.9	70.8	25.7	49.9	59.5	66.9	72.1
CRRRO-BRO [4]	23.3	54.0	69.8	76.7	82.7	9.9	26.1	40.2	54.2	64.9	17.8	34.6	48.1	57.5	62.2
Covariance [5]	26.5	52.5	66.0	80.0	90.9	21.3	43.6	60.4	70.3	78.2	38.0	61.0	73.1	79.0	82.5
PREF [7]	30.6	55.3	67.0	82.0	92.6	22.3	44.3	58.5	67.4	74.4	43.0	68.7	77.9	82.2	85.2
BSC+CM [8]	32.0	59.1	72.3	82.4	93.1	23.1	44.3	56.4	64.3	70.4	58.6	80.6	87.4	90.4	92.1
TriNet (local) [12]	25.0	53.2	65.6	78.2	84.4	37.1	57.3	66.3	71.9	79.9	67.8	87.7	88.9	93.8	96.3
AlignReID (local) [26]	28.1	56.3	68.8	75	87.5	32.6	51.2	59.6	66.3	71.2	69.8	87.4	94.1	94.1	96.3
TriNet (global) [12]	33.6	55.0	69.4	77.5	86.9	23.6	42.7	60.7	69.7	74.2	34.6	65.4	82.7	82.4	90.2
AlignReID (global) [26]	34.4	62.5	75.0	84.3	93.7	18.0	43.8	55.1	66.3	77.5	39.5	55.6	70.4	77.8	85.9
Proposed-auto (hand)	37.9	64.5	79.4	91.5	93.8	47.4	68.1	77.3	83.6	87.4	72.3	90.6	94.1	97.1	97.5
Proposed-auto (conv)	38.8	65.7	82.5	93.8	98.8	48.4	75.2	89.9	93.3	94.4	80.2	93.8	96.3	97.5	97.5

Table 7. Table VII: Running time on the three datasets

Datasets	i-LIDS MCTS	DukeMTMC Group	Road Group
All image pairs	1.1 min	18.9 min	11.5 min
Per image pair	0.06 sec	0.14 sec	0.10 sec

Equations

$$L_{ij}(k)=\mathcal{N}(k-m;0,{\sigma }_L)$$

$$P_{ij}(k)=\mathcal{N}(k-m_{ij};0,{\sigma }_P)+\mathcal{N}(k-m_{ij};\pm 9,{\sigma }_P)$$

$${\mathbf{f}}_{(i,j)}^s=[L_{ij},P_{ij}]$$

$${\mathbf{f}}_{i,m}^l=tanh({\mathbf{W}}_l{\hat{\mathbf{f}}}_{i,m}^l)$$

$${\mathbf{f}}_{(i,j),m}^s=tanh({\mathbf{W}}_s[{\hat{\mathbf{f}}}_{i,m}^l,{\hat{\mathbf{f}}}_{j,m}^l])$$

$$L_g=[d_2({\mathbf{f}}_a^g,{\mathbf{f}}_p^g)-d_2({\mathbf{f}}_a^g,{\mathbf{f}}_n^g)+{\lambda }_g{]}_{+}$$

$$L_a=\frac{1}{|{\mathcal{O}}_{1,a}|}\sum _{i\in {\mathcal{O}}_{1,a}}{\left[d_2({\mathbf{f}}_{i,a}^l,{\mathbf{f}}_{i^{\prime },p}^l)-d_{neg}(i)+{\lambda }_l\right]}_{+}$$

$$d_{neg}(i)=\frac{1}{|\mathcal{K}|}\left(\sum _{j\in {\mathcal{K}}_p}{d}_2({\mathbf{f}}_{i,a}^l,{\mathbf{f}}_{j,p}^l)+\sum _{j\in {\mathcal{K}}_n}{d}_2({\mathbf{f}}_{i,a}^l,{\mathbf{f}}_{j,p}^l)\right)$$

$$[{\hat{\delta }}_{(i,j),m}^x,{\hat{\delta }}_{(i,j),m}^y{]}^T=W_r{\mathbf{f}}_{(i,j),m}^s$$

$${\delta }_{(i,j),m}^x=\frac{x_{j,m}-x_{i,m}}{w_{i,m}}$$

$${\delta }_{(i,j),m}^y=\frac{y_{j,m}-y_{i,m}}{h_{i,m}}$$

$$L_s=\frac{1}{P}\sum _{m\in \{a,p,n\}}\sum _{(i,j)\in {\mathcal{O}}_{2,m}}\sum _{t\in \{x,y\}}S\left({\hat{\delta }}_{(i,j),m}^t,{\delta }_{(i,j),m}^t\right)$$

$$L=L_g+{\lambda }_1{L}_a+{\lambda }_2{L}_s$$

$${\alpha }_i=t_1(i,{\mathcal{G}}_{\backslash i})+s(i,{\mathcal{M}}_i)+p({\mathcal{M}}_i,{\mathcal{M}}_{{\mathcal{G}}_{\backslash i}}),$$

$$t_1(i,{\mathcal{G}}_{\backslash i})=\sum _{i^{\prime }\in {\mathcal{G}}_{\backslash i}}\frac{{\rho }_i}{{\rho }_{i^{\prime }}},$$

$$s(i,{\mathcal{M}}_i)={\lambda }_s\frac{d_f({\mathbf{f}}_i,{\mathbf{f}}_{{\mathcal{M}}_i})}{|{\mathcal{M}}_i|}.$$

$${\lambda }_s=\frac{1}{{\sum }_i{d}_f({\mathbf{f}}_i,{\mathbf{f}}_{{\mathcal{M}}_i})/|{\mathcal{M}}_i|}$$

$$p({\mathcal{M}}_i,{\mathcal{M}}_{{\mathcal{G}}_{\backslash i}})=\sum _{i^{\prime }\in {\mathcal{G}}_{\backslash i}}{\lambda }_p{d}_m({\mathcal{M}}_i,{\mathcal{M}}_{i^{\prime }}),$$

$${\alpha }_{i_1{i}_2}={\alpha }_{i_1}+{\alpha }_{i_2}+t_2(i_1,i_2).$$

$${\alpha }_{i_1{i}_2{i}_3}={\alpha }_{i_1{i}_2}+{\alpha }_{i_2{i}_3}+{\alpha }_{i_1{i}_3}+t_3(i_1,i_2,i_3).$$

$$t_3(i_1,i_2,i_3)=\exp \left(-2\ast \sum _{k=1}^3|\sin {\theta }_k-\sin \frac{\pi }{3}|\right)$$

$$\mathcal{Q}(\mathcal{C})=$$

$$+\sum _{r\ne l,r,l\in \{1,2,3,g\}}{\mathcal{P}}_{rl}(\mathcal{C}),$$

$${\mathcal{P}}_1(\mathcal{C})=\text{leavevmode}\sum _{c_{ij}\in \mathcal{C}}{m}_1(c_{ij})=\text{leavevmode}\sum _{c_{ij}\in \mathcal{C}}{w}_1({\mathbf{f}}_i,{\alpha }_i,{\mathbf{f}}_j,{\alpha }_j)$$

$$m_1(c_{ij})=w_1({\mathbf{f}}_i,{\alpha }_i,{\mathbf{f}}_j,{\alpha }_j)={\lambda }_{w_1}\frac{\psi ({\alpha }_i,{\alpha }_j)}{d_{\mathbf{f}}({\mathbf{f}}_i,{\mathbf{f}}_j)}$$

$$\displaystyle\mathcal{P}_{2}(\mathcal{C})=\leavevmode\nobreak\$$

$$\displaystyle=\leavevmode\nobreak\$$

$$w_2({\mathbf{f}}_{i_1{i}_2},{\alpha }_{i_1{i}_2},{\mathbf{f}}_{j_1{j}_2},{\alpha }_{j_1{j}_2})={\lambda }_{w_2}\frac{\psi ({\alpha }_{i_1{i}_2},{\alpha }_{j_1{j}_2})}{d_{\mathbf{f}}({\mathbf{f}}_{i_1{i}_2},{\mathbf{f}}_{j_1{j}_2})}$$

$$\displaystyle\mathcal{P}_{3}(\mathcal{C})=\leavevmode\nobreak\$$

$$\displaystyle=\leavevmode\nobreak\$$

$$\displaystyle\mathcal{P}_{g}(\mathcal{C})=\leavevmode\nobreak\$$

$$\displaystyle=\leavevmode\nobreak\$$

$$\displaystyle\mathcal{P}_{rl}(\mathcal{C})=\leavevmode\nobreak\$$

Full text

I Introduction

Person re-identification (Re-ID) aims at matching and identifying pedestrians across non-overlapping camera views. This task is increasingly important in visual surveillance and has attracted much attention in recent research [1, 2, 3]. However, most research focused on individual person Re-ID, while the Re-ID of groups of people are seldom studied. In practice, since most events (e.g., moving, fighting or violent actions) could be performed within distinct groups instead of between individuals, it is essential to identify groups rather than single persons when analyzing events across cameras [4, 5]. Therefore, it is non-trivial to obtain reliable group matching across different camera views.

In group Re-ID, there are two more basic challenges besides viewpoint changes and human pose variations [6, 3], both inherent issues for individual person case. These challenges are as follows: (i) Group layout change: The layout of people in a group are largely unconstrained across different camera views. Due to the dynamic movements of people, the relative positions of people in a group may have large differences in two camera views (cf. Fig. 1a). (ii) Group membership change: people may often join or leave a group (cf. Fig. 1b).

Most existing methods, e.g., [4, 5, 7], view the input group image as an entire unit and extract global/semi-global features without explicitly doing individual people matching and considering layout changes to perform group-wise matching [4, 5]. A recent study [8] attempts to use descriptors of local patches to partially handle layout change and membership change.

In this paper, we introduce the idea of group granularity and characterize a group image by multi-grained objects. By defining crowds with large membership overlap as of the same group, a group can be depicted with multi-grained objects:

fine-grained objects are formed by a single person, medium-grained objects are formed by a group of two people, coarse-grained objects are formed by a group of three people, and global-grained object consists of all persons in the group. We argue that characterization of objects from multiple granularities is helpful to enhance the invariance of descriptors towards changes in group membership and layout.

We refer to the example in Fig. 1a for better clarity on the need for group granularity. Due to the large layout variation and camera viewpoint changes, the same group shows large global appearance differences in two camera views. The Re-ID performance is poor if global features are merely adopted for the entire group for re-identification (cf. bottom-right image in Fig. 1a). This issue can be resolved if we include information of finer group granularity (e.g., individual persons). On the other hand, merely using the information of individual people is also not always reliable. An example is shown in Fig. 1b where two groups in Camera $B$ which contain visually similar group members to the probe group in Camera $A$. In this case, the information at medium-level granularity (e.g., subgroups of two people) would be useful.

Our approach leverages on the representations of multiple granularities, also termed as multi-grained objects (cf. Fig. 1c), for group re-identification. In addition, motivated by the observations that groups in different cameras may be interfered by group member variation, occlusion, and mismatching, and that multi-grained objects have different reliabilities on Re-ID performances, we propose to introduce the dynamic updated importance weights to explicitly model the different characterization power of each objects in different granularity and further improve the group re-identification performance.

Meanwhile, due to the strong ablility of convolutional neural networks to extract local invariant features, some deep learning based methods have achieved unprecedented performance in vision recognition tasks in recent years [9, 10]. Inspired by these works, many deep learning techniques have been applied to the person Re-ID task [11, 12] and are demonstrated to outperform some traditional pipelines, which are typically reliant on manually designed feature representations and metric learning algorithms. Nevertheless, few works utilize deep learning methods for group re-identification. Considering the large variation in illumination, membership change in crowds and pose transformation in group Re-ID dataset, there is sufficient motivation to study the performance of deep CNNs on the group Re-ID task.

For comprehensiveness, we introduce two independent pipelines for feature extraction. One is a combination of traditional handcrafted algorithms while the other is based on a multi-task convolutional neural network. Both pipelines extract their own set of appearance and spatial relation features of different granularities. Our experiments convincingly show that our group Re-ID framework is able to achieve state-of-the-art results on different datasets with either hand-crafted features or deep convolutional features.

In summary, our contributions are four-fold:

We introduce multi-grained representations for group images to better handle changes in group layout and membership, coupled with a dynamic weighting scheme for better person matching. 2. 2.

We solve the group-wise matching problem by using a multi-order matching algorithm that integrates multi-grained representations and combines the information of both matched and unmatched objects to achieve a more reliable matching result. 3. 3.

We propose two schemes to extract appearance and spatial relation features for the multi-grained representation: one based on typical hand-crafted features, the other on deep CNN features. For the latter, a new multi-task integrated CNN is designed for this specific purpose. 4. 4.

We create two challenging group Re-ID datasets with large group membership and layout variations. The existing group Re-ID datasets consists of relatively small group sizes and group layout, which are less realistic in real-world scenarios.

II Related Works and Overview

Person Re-ID has been studied for years. Most of them focus on developing reliable features [13, 14], deriving accurate feature-wise distance metric [15, 16], and handling local spatial misalignment between people [3, 17]. Some recent research works extend Re-ID algorithms to more object types (e.g., cars [18]) or more complex scenarios (e.g., larger camera numbers [19], long-term videos [20], untrimmed images [21, 22]).

During recent years, some deep learning based methods have emerged to solve the problem of single person Re-ID. These methods have a strong ability to extract rich invariant features from images. A number of works [23, 24] exploit CNNs for person Re-ID by exploiting pairwise labels from positive and negative sample pairs in a variety of network architectures. More recent works [25, 11, 26] are inclined to equip CNNs with triplet loss [27], which has shown to perform exceedingly well [12, 28] by learning a representative feature embedding space which facilitates a distance metric.

However, most existing works focus on the Re-ID of individual person; as such, the group-level Re-ID problem is seldom considered. Since group Re-ID contains significant group layout changes and group membership variations, it introduces new challenges and a proliferation of information that requires encoding as compared to scenarios addressed by single person Re-ID methods. Although some works [29, 30] introduce people or group interaction into Re-ID process, they are only targeted at improving the Re-ID performance of a single person. The characteristics of groups are still less considered and not fully modeled.

Only a few works have been proposed to address group Re-ID tasks [4, 5, 8, 7]. Most of them develop global or semi-global features to perform group-wise matching. For example, Cai et al. [5] proposed a discriminative covariance descriptor to capture the global appearance & statistic properties of group images. Zheng et al. [4] segmented a group image into multiple ring regions and derived semi-global descriptors for each region. Lisanti et al. [7] combined sparsity-driven descriptions of all patches into a global group representation. Since global or semi-global features are unlikely to capture information of local interaction in groups, they may have limitations in handling complex scenarios with significant group appearance variations caused by pose and background interference.

Recently, Zhu et al. [8] developed a local-based method which performs group Re-ID by selecting proper patch-pairs and conducting patch matching between cross-view group images. However, in order to reduce patch mismatches, this method includes prior restrictions on vertical misalignments. This limits their capability in handling significant group layout changes or group member variations.

Our approach differs from the existing group Re-ID works in two aspects: (1) The existing works perform Re-ID with information derived from single granularity (i.e., either global or patch level information). Comparatively, our approach leverages multi-grained information to fully capture the characteristics of a group. (2) Our approach does not include any prior restrictions on spatial misalignments, which is able to handle arbitrary group layout changes or group member variations.

Overview of our approach. Given the probe group image captured from one camera, our goal is to find the matched group images from a set of gallery group images captured from another camera. We represent each group image by a set of multi-grained objects, and then proceed to extract features by a combination of hand-crafted descriptors, or by a forward pass on a multi-task CNN. With these features, the matching process computes the static and dynamic importance weights of multi-grained objects between the probe and gallery images. Then, a multi-order matching algorithm computes intermediate matching results, which are used to update the dynamic importance weights. We perform these two stages in iterative fashion, with final matching results obtained at convergence. The entire framework is shown in Fig. 2.

III Multi-grained Representation

A group image $I$ contains a set of people: $\mathcal{B}=\{b_{1},b_{2},\dots,b_{N}\}$, where $N$ is the number of people and $b_{i}$ (or simply denoted by $i$ for presentation clarity) corresponds to the person bounding box. The representation is computed by building multi-grained objects (people/subgroups):

1. Fine granularity, including objects of an individual person, $\mathcal{O}_{1}=\{i|i=1,\dots,N\}$;
2. Medium granularity, including objects of two-people subgroups, $\mathcal{O}_{2}=\{(i_{1},i_{2})|i_{1},i_{2}=1,\dots,N,i_{1}\neq i_{2}\}$;
3. Coarse granularity, including objects of three-people subgroups, $\mathcal{O}_{3}=\{(i_{1},i_{2},i_{3})|i_{1},i_{2},i_{3}=1,\dots,N,i_{1}\neq i_{2}\neq i_{3}\}$; and 4) Global granularity, referring to the entire group, $\mathcal{O}_{g}=\{(1,2,\dots,N)\}$. In the cases where there are only two people in the group image, we simply let $\mathcal{O}_{2}$ be the coarse granularity.

Choice of granularities. We adopt four levels of granularity because the combination of three distinct levels and a global level is sufficient to characterize both the global appearance and local layout of crowds, besides for tractability reasons. The fine granularity helps reduce the confusion in the global appearance when encountering large layout or group member changes, while the medium and coarse granularities can help resolve ambiguous individual person matches in the fine granularity by incorporating local layout or co-occurrence information in a group.

Feature Notations. The feature of an object $o\in\mathcal{O}_{1}$ in the fine granularity, denoted by $\mathbf{f}_{o}=\mathbf{f}^{l}_{o}$, is about the local appearance. The feature of an object $o\in\mathcal{O}_{2}\cup\mathcal{O}_{3}$ in the medium and coarse granularity, denoted by $\mathbf{f}_{o}=[\mathbf{f}^{l}_{o},\mathbf{f}^{s}_{o}]$, consists of two parts:

1. appearance, which is an aggregation of features representing the local appearances of all people within the subgroup, and; 2) spatial relation, a single or aggregation of features $\mathbf{f}^{s}_{o}$ that describes the spatial layout of each edge linking two different individuals within this subgroup. Here, aggregation indicates concatenating the feature of the same granularity and semantics, followed by performing t-SNE [31] for feature reduction. The notation $[\cdot,\cdot]$ denotes a vector concatenation operation, which also applies to the rest of this paper. Meanwhile, the global feature of object $o\in\mathcal{O}_{g}$, denoted by $\mathbf{f}_{o}=\mathbf{f}^{g}_{o}$, describes the appearance feature of the whole input group image.

As shown in Fig. 2, our framework is independent of the choice of feature vectors used, which implies that we could exploit some conventional algorithms to extract hand-crafted features. On the other hand, we also intend to exploit the strong ability of deep CNNs in extracting more representative and invariant features. Therefore, we introduce two different pipelines to obtain these feature representations for usage in our group Re-ID framework. The first extracts a combination of manually designed descriptors to encode both object appearances and layout relationship between objects while the second is a new integrated deep learning based method to extract features in a single forward pass.

III-A Hand-crafted Feature Descriptors

In this section, we briefly describe the hand-crafted features utilized in this work. Color and texture features [15] are used as the appearance part of the object. To be specific, for each single person input image (obtained from bounding box $b_{i}$ and resized to unified resolution), we split it into $18$ equal-sized patches along the vertical direction, and the RGB, HSV, YCbCr, LAB, YIQ color features and Gabor texture features are extracted from each patch. The output normalized histograms of these features are concatenated to form a final $8,024$-dimensional local appearance descriptor $\mathbf{f}^{l}_{o}$. Similarly, we also extract global features $\mathbf{f}^{g}_{o}$ in the same manner but with the input image containing the entire group.

As for the spatial relation part, we use the relative distance & angle histograms among individuals in an object [32] to describe each edge between two people ($i$,$j$). For two bounding boxes $b_{i}$, $b_{j}$ within a subgroup, we first denote the relative position between their centers in polar coordinate $(\rho_{ij},\theta_{ij})$, where $\rho_{ij}$ is the log-distance between the two centers and $\theta_{ij}$ is the corresponding orientation angle. The 10-D log-distance histogram $L_{ij}$ and 9-D angle histogram $P_{ij}$ are constructed over uniform bins as follows:

[TABLE]

[TABLE]

where $\mathcal{N}(x;\mu,\sigma)$ is a discrete Gaussian window parameterized by mean $\mu$ and variance $\sigma$, while $m_{ij}$ is the index of bin containing $\rho_{ij}$ or $\theta_{ij}$. Finally, the two output histograms are combined to form the descriptor that represents the $b_{i}$-$b_{j}$ edge:

[TABLE]

III-B Integrated CNN for Feature Extraction

Inspired by modern CNN based object detection frameworks  [33], which perform exceedingly well in both recognition and localization tasks, we hypothesize that deep neural networks can be tailored to handle both appearance and structural layout of objects in an integrated manner. As such, we propose a new multi-task network to jointly extract both appearance and spatial relation features needed for our group Re-ID framework.

The overview of the multiple-task CNN in our work is depicted in Fig. 3a. For each input image, we use the ResNet-34 [9] as the backbone structure for basic feature extraction. Post-processing includes two separate branches: The global branch, which is responsible for extracting features from the entire group image, and a local branch, which is utilized to handle individual objects and their relations to others. With these branches processed in parallel, we could obtain the multi-grained features simultanously.

III-B1 Minibatch organization

We borrow the idea of using triplet loss for training [12, 11, 34] to learn representative mapping of images to an abstract feature space. Therefore, we organize the minibatch into triplets where each training sample consists of three images: an anchor image $I_{a}$ from probe images, a positive image $I_{p}$ from the gallery which contains the same group as anchor, and a negative image $I_{n}$ which is randomly selected from training set that has different group id from the anchor and positive images.

Throughout this section, we shall adopt the same subscripts to denote anchor, positive and negative images for features or object sets. For example, $\mathbf{f}^{g}_{a}$ denotes global feature of the anchor image and $\mathcal{O}_{1,a}$ denotes the set of person objects in the anchor image.

III-B2 Global feature extraction

The global branch receives feature maps from the final convolution layer of the 5-th stage of ResNet-34, which is denoted as S5 in Fig. 3a. We then apply a simple global average pooling operation followed by a fully-connected layer as the global feature extraction module to obtain the corresponding global features $\mathbf{f}_{m}^{g}$, $m\in\{a,p,n\}$.

III-B3 Local feature extraction

The local branch receives feature maps from the final convolution layer of the 4-th stage of ResNet-34 (denoted as S4 in Fig. 3a) and bounding box sets $\mathcal{B}_{a},\mathcal{B}_{p}$ and $\mathcal{B}_{n}$ belonging to each image of the triplet. The local feature extraction module applies ROI Pooling [33] on each feature map according to its bounding box and sends the output to a fully-connected layer. Hence, we obtain the intermediate local features $\hat{\mathbf{f}}_{i,m}^{l}$ for each $i$-th individual person in the image of set $m\in\{a,p,n\}$.

Next, these features are further utilized in two sub-branches. First, a feature projection module takes the intermediate features as input and yields local appearance features by nonlinear projection that constrains outputs to (-1, 1):

[TABLE]

where $\mathbf{W}_{l}$ is a learnable transformation matrix. Secondly, to model the spatial relation between two objects, i.e. an edge linking two individuals ($i$,$j$), a feature fusion module is applied to fuse intermediate features with another learnable transformation matrix $\mathbf{W}_{s}$ as follows:

[TABLE]

III-B4 Loss function

To regularize the global and local output features from different branches, we design three different types of losses to train our multi-task CNN. For the global appearance features, we intend to learn an embedding space where the anchor sample will be closer to the positive sample than the negative sample. For this, we obtain a group-wise training loss by utilizing the triplet loss:

[TABLE]

where $[\cdot]_{+}$ is the ReLU operator $\max(0,\cdot)$, $d_{2}(\cdot,\cdot)$ denotes the L2-norm distance between the two feature vectors, and $\lambda_{g}$ is a hyper-parameter to control the margin size.

For the supervision of individual appearance features, we could simply apply a triplet loss over the matched and unmatched pairs, similar to that in Eq. 6. However, since the number of matched individual pairs between anchor and positive images is much lesser than that of unmatched ones, the loss might well be dominated by unmatched pairs. Inspired by the Trihard loss [12], we impose the hard negative mining strategy to the standard triplet loss, much akin to a k-nearest negative neighbors manner:

[TABLE]

where $i^{\prime}\in\mathcal{O}_{1,p}$ is the individual person in the positive image who matches exactly to the $i$-th person in the anchor image. Note that if person $i$ in anchor does not match any individual in positive image, we set the $d_{2}(\cdot,\cdot)$ term in Eq. 7 to be zero. The hyper-parameter $\lambda_{l}$ controls the margin size of local appearance features. The term $d_{neg}(i)$ is the average distance between person $i$ in anchor image and persons in set $\mathcal{K}=\mathcal{K}_{p}\cup\mathcal{K}_{n}$, which carries the intuition of $k$-nearest unmatched individuals from other images in the feature space:

[TABLE]

where $\mathcal{K}_{p}$ and $\mathcal{K}_{n}$ denotes the collection of $k$ nearest unmatched individuals from positive and negative images respectively.

In the work of [33], deep neural networks can precisely predict the relative offset between two bounding boxes. Based on this observation, we design a regression loss to supervise the learning of spatial relation features. Given the bounding boxes, $b_{i,m},b_{j,m}\in\mathcal{B}_{m}$ (with $m\in\{a,p,n\}$ as mentioned before) of two individuals ($i$,$j$) in an image and the feature $\mathbf{f}_{(i,j),m}^{s}$ representing the edge between them, we apply linear regression with learnable parameter $W_{r}$ to predict the normalized spatial transition:

[TABLE]

Suppose the bounding box is in the form of $b_{m,i}=(x_{i,m},y_{i,m},w_{i,m},h_{i,m})$ as defined in [33], we denote the ground truth transition as:

[TABLE]

Hence, the localization loss over spatial relation features is:

[TABLE]

where $P=\sum_{m\in\{a,p,n\}}|\mathcal{O}_{2,m}|$ is the sum of total number of bounding box pairs within each image of the triplet. $S(\cdot,\cdot)$ denotes the smooth L1 loss used also in [33]. Finally we combined the losses in Eq. 6, Eq. 7 and Eq. 12 to obtain the final objective function for our multi-task CNN framework:

[TABLE]

where $\lambda_{1}$, $\lambda_{2}$ are balancing factors. After the training phase, this multi-task network is used to extract deep multi-grained features in a single forward pass. Aggregation of these local features is performed to attain features of coarse and medium granularities for the subsequent matching step. In our experiments, we set $\lambda_{g}$ and $\lambda_{l}$ as $2.0$, and the balancing factors $\lambda_{1}$ and $\lambda_{2}$ as $1.0$ during training.

IV Importance Weighting

We introduce an importance weight $\alpha_{o}$ for each object $o$ (except the global-grained object) to indicate the object’s discriminativity and reliability inside the group image for group person matching. The importance weighting scheme is partially inspired by but different from the saliency-learning methods [6, 8] for differentiating patch reliabilities in person re-identification: (i) Our scheme aims to weight each granularity object rather than patches; (ii) Our scheme dynamically adjusts the importance weights in an iterative manner, by using the intermediate matching results at each iteration. (cf. Fig. 2).

IV-A Fine-grained Object

The importance weight ($\alpha_{i}$) for each individual person $i$ in the probe image $I$ consists of two components: static weight, which is only dependent on the group image, and dynamic weight, which is dynamically updated according to the intermediate matching results with the gallery group images, from another camera in our approach. The formulation of this weight is given as follows,

[TABLE]

where the first term is the static weight, and the second and third terms form the dynamic weight.

Static weight. The static weight $t_{1}(i,\mathcal{G}_{\backslash i})$, where $\mathcal{G}_{\backslash i}=\mathcal{G}-\{i\}$ denotes the set of other individual people in $\mathcal{G}$, is used to describe the stability. It is computed as follows,

[TABLE]

where $\rho_{i}$ is the local density around person $i$ in group $\mathcal{G}$. It reflects the density of people in a neighborhood around $i$, which is computed by following [35].

By Eq. 15, the static weight $t_{1}$ is mainly obtained by evaluating the relative local density ratios between person $i$ and his/her peer group members ${i{{}^{\prime}}}$ in $\mathcal{G}$. If the local density around $i$ is larger than the density around his/her peer group members ${i{{}^{\prime}}}$, the stability of $i$ is increased, indicating that $i$ is located in the center region of group $\mathcal{G}$ and should be a more reliable member in group Re-ID (cf. person $1$ in Fig. 4a). On the contrary, when $i$’s local density is smaller than his/her peer group members, a small stability value will be assigned, indicating that $i$ is located in the outlier region of the group and is less reliable (cf. person $2$ in Fig. 4a).

Dynamic weight. The dynamic weight $s(i,\mathcal{M}_{i})+p(\mathcal{M}_{i},\mathcal{M}_{\mathcal{G}_{\backslash i}})$ consists of two parts: the saliency term $s(i,\mathcal{M}_{i})$ and the purity term $p(\mathcal{M}_{i},\mathcal{M}_{\mathcal{G}_{\backslash i}})$, where $\mathcal{M}_{i}$ is the set of matches from the gallery group images, and $\mathcal{M}_{\mathcal{G}_{\backslash i}}$ is the set of the matches for all people except $i$ in the probe image, $\mathcal{M}_{\mathcal{G}_{\backslash i}}=\{\mathcal{M}_{i^{\prime}}|i^{\prime}\notin\mathcal{G}\}$. The sets of matches are illustrated in Fig. 4a.

The saliency term is computed as

[TABLE]

Here $d_{f}(\cdot)$ is the Euclidean distance between features. $|\mathcal{M}_{i}|$ is the cardinality of $\mathcal{M}_{i}$. $\mathbf{f}_{\mathcal{M}_{i}}$ is the feature describing the set of matches $\mathcal{M}_{i}$, and we use the feature of the $\frac{1}{2}|\mathcal{M}_{i}|$th nearest neighbor of $i$ in $\mathcal{M}_{i}$ as done in [36, 6]. $\lambda_{s}$ is an adaptive normalization factor to normalize the range of ${s}$ to be within [math] to $1$, which is computed as follows:

[TABLE]

For simplicity, the other normalization factors in the rest of this paper are calculated in the similar way as $\lambda_{s}$.

According to Eq. 16, if the appearance of an individual person $i$ is discriminative, a large portion of individuals in $i$’s matched set $\mathcal{M}_{i}$ are visually dissimilar to $i$. This leads to a large $d_{f}(\mathbf{f}_{i},\mathbf{f}_{\mathcal{M}_{i}})$ and a large saliency value [36, 6] (cf. person $1$ in Fig. 4a). Moreover, due to the variation of group members in group Re-ID, each individual person may have different number of matched people in his/her $\mathcal{M}_{i}$. Therefore, we further introduce $|\mathcal{M}_{i}|$ in Eq. 16, such that person with fewer matched people can indicate more discriminative appearance.

The purity term is computed as

[TABLE]

where $d_{m}(\cdot)$ is the Wasserstein-$1$ distance [37], a measure to evaluate the dissimilarity between two feature sets. $\lambda_{p}$ is calculated in the same way as $\lambda_{s}$ in Eq. 16.

According to Eq. 18 and Fig. 4a, the purity measurement reflects the relative appearance uniqueness of person $i$ inside group $\mathcal{G}$. If $i$ has similar appearance features as other group members in $\mathcal{G}$, their matched people in camera $B$ should also be visually similar and located close to each other in the feature space (cf. $\mathcal{M}_{3}$ and $\mathcal{M}_{4}$ in Fig. 4a), resulting in a small purity value. On the other hand, if a person includes unique appearance features in $\mathcal{G}$, his/her matched people in camera $B$ should have larger feature distances to those of the other members in $\mathcal{G}$, and lead to a large purity value (cf. $\mathcal{M}_{1}$ in Fig. 4a).

IV-B Medium and Coarse grained Objects

The importance weight $\alpha_{i_{1}i_{2}}$ of a medium-grained object $(i_{1},i_{2})$ is computed as:

[TABLE]

Here, ${t}_{2}(i_{1},i_{2})$ is the stability measure of the sub-group $(i_{1},i_{2})$. A two-people sub-group is thought more stable if its members are spatially closer to each other. Thus, we simply compute ${t}_{2}$ by the inverse of spatial distance between $i_{1}$ and $i_{2}$.

The importance weight $\alpha_{i_{1}i_{2}i_{3}}$ of a coarse-grained object $(i_{1},i_{2},i_{3})$ is computed as:

[TABLE]

Here, $\alpha_{i_{1}i_{2}}$ is the importance of a two-people pair in $(i_{1},i_{2},i_{3})$ (cf. Eq. 19). ${t}_{3}$ is the stability of a three-people subgroup. We assume equilateral triangle as the most stable structure for three-people subgroups and model ${t}_{3}$ by evaluating its similarity to a equilateral triangle according to Eq. 21, where $\theta_{k},k\in\{1,2,3\}$ denote the angles of the triangle constructed by coarse sub-group $(i_{1},i_{2},i_{3})$.

[TABLE]

Fig. 4b shows the importance weights of some groups. From Fig. 4b, we can see that our process can effectively set larger weights on objects with stronger characterization ablility to represent the entire group.

IV-C Iterative Update

We utilize an iterative process which updates the importance weights and group-wise matching results iteratively. We initialize the dynamic weights for all objects by $1$ and compute the optimal matching through multi-order matching (cf. Sec. V) to obtain an initial matching result: $\mathcal{M}_{1},\mathcal{M}_{2},\dots,\mathcal{M}_{N}$. This matching result is used to update the dynamic importance weights. This procedure is repeated until the importance weights become converged or the maximum iteration is reached. Although the exact convergence of our iterative process is difficult to analyze due to the inclusion of multi-order matching, in our experiments we confirm that most importance weights become stable within $5$ iterations, which implies the reliability of our approach.

V Multi-order Matching

Given a probe image $I_{p}$ and a gallery image $I_{g}$, our goal is to compute the matching score between the two groups of people. Suppose that there are $N_{p}$ people in the probe image $I_{p}$ and $N_{g}$ people in the gallery image $I_{g}$. The goal of the multi-order matching process aims to find: (1) an optimal one-to-one mapping, $\mathcal{C}=\{(i,j)|\leavevmode\nobreak\ \forall(i,j),(i{{}^{\prime}},j{{}^{\prime}}),i\neq i{{}^{\prime}},j\neq j{{}^{\prime}}\}$, where $(i,j)$ ($=c_{ij}$) denotes a match between the $i$th person from the probe image and the $j$th person from the gallery image, and (2) the maximum matching score.

Since a group is characterized by multiple granularities, it is natural to measure the similarity across different granularities to find the optimal match. With this consideration, The objective function of our matching process is formulated with multi-order potentials:

[TABLE]

where $\mathcal{P}_{1}(\mathcal{C})$, $\mathcal{P}_{2}(\mathcal{C})$, $\mathcal{P}_{3}(\mathcal{C})$, and $\mathcal{P}_{g}(\mathcal{C})$ are the first-order, second-order, third-order, and global potentials, evaluating the matching quality over each subgroup of people, and $\mathcal{P}_{rl}(\mathcal{C})$ is the inter-order potential.

V-A Multi-order Potentials

First order potential. $\mathcal{P}_{1}(\mathcal{C})$ is used to model the matching scores over individual people. It is calculated by the sum of the matching scores of all the individual matches in $\mathcal{C}$:

[TABLE]

where $\mathbf{f}_{i}$, $\alpha_{i}$ and $\mathbf{f}_{j}$, $\alpha_{j}$ are the feature vector and importance weight for probe-image person $i$ and gallery-image person $j$, respectively (cf. Eq. 14). $m_{1}(c_{ij})=w_{1}(\mathbf{f}_{i},\alpha_{i},\mathbf{f}_{j},\alpha_{j})$ is the matching score for match $c_{ij}=(i,j)$, calculated by:

[TABLE]

where $\psi(\alpha_{i},\alpha_{j})=\frac{\alpha_{i}+\alpha_{j}}{1+|\alpha_{i}-\alpha_{j}|}$ is the fused importance weight, which will have large value if the importance weights of $\alpha_{i}$ and $\alpha_{j}$ are both large and close to each other. $d_{\mathbf{f}}(\cdot)$ is the Euclidean distance and $\lambda_{w_{1}}$ is the normalization constant for the first-order potential.

By Eq. 24, the matching score $m_{1}(c_{ij})$ is computed by the importance-weighted feature similarity $w_{1}(\mathbf{f}_{i},\alpha_{i},\mathbf{f}_{j},\alpha_{j})$ between the matched individuals $i$ and $j$.

Second order potential. $\mathcal{P}_{2}(\mathcal{C})$ is used to model the matching scores over two-people subgroups:

[TABLE]

where $\mathbf{f}_{i_{1}i_{2}}$, $\alpha_{i_{1}i_{2}}$ and $\mathbf{f}_{j_{1}j_{2}}$, $\alpha_{j_{1}j_{2}}$ are the feature vector and importance weight for probe-image subgroup $(i_{1},i_{2})$ and gallery-image subgroup $(j_{1},j_{2})$, respectively (cf. Eq. 19). $m_{2}(c_{i_{1}j_{1}},c_{i_{2}j_{2}})=w_{2}(\mathbf{f}_{i_{1}i_{2}},\alpha_{i_{1}i_{2}},\mathbf{f}_{j_{1}j_{2}},\alpha_{j_{1}j_{2}})$ is the second order match score between two-people subgroups $(i_{1},i_{2})$ and $(j_{1},j_{2})$, which is calculated in a similar way as Eq. 24:

[TABLE]

Third order potential. $\mathcal{P}_{3}(\mathcal{C})$ is used to model the matching scores over three-people subgroups:

[TABLE]

where the term $w_{3}(\mathbf{f}_{i_{1}i_{2}i_{3}},\alpha_{i_{1}i_{2}i_{3}},\mathbf{f}_{j_{1}j_{2}j_{3}},\alpha_{j_{1}j_{2}j_{3}})=m_{3}(c_{i_{1}j_{1}},c_{i_{2}j_{2}},c_{i_{3}j_{3}})$ is the third order match score between three-people subgroups $(i_{1},i_{2},i_{3})$ and $(j_{1},j_{2},j_{3})$. It is calculated in the same way as Eq. 26.

Global potential. The global potential is calculated by the global matching score between probe and gallery images $I_{p}$ and $I_{g}$:

[TABLE]

where $\mathbf{f}_{p}$ and $\mathbf{f}_{g}$ are the global feature vectors for the entire group images $I_{p}$ and $I_{g}$. $\alpha_{p}=\alpha_{g}=1$ are the importance weights for global objects. In this paper, we simply use the global feature similarity as the global matching score, as: $w_{g}(\mathbf{f}_{p},\alpha_{p},\mathbf{f}_{g},\alpha_{g})=\frac{1}{d_{\mathbf{f}}(\mathbf{f}_{p},\mathbf{f}_{g})}$.

Inter-order potential. Since each match $c_{ij}$ is described by potentials in multiple orders (cf. Eqs. 23-28), we also introduce inter-order potentials to properly combine these multi-order potential information. Specifically, the inter-order potential between orders $r,l\in\{1,2,3,g\}$ is calculated by:

[TABLE]

where $m_{rl}(c_{ij},r,l)$ is the inter-order correlation for match $c_{ij}$. It is calculated by:

[TABLE]

where $\lambda_{k}$ is the normalization constant for order $k$, $m_{k}(c_{i^{{}^{\prime}}_{1}j^{{}^{\prime}}_{1}}...c_{i^{{}^{\prime}}_{k}j^{{}^{\prime}}_{k}})$ is the intra-order match score in order $k$ (as in Eqs. 24 and 26). From Eq. 30, if a match $c_{ij}$ creates large and similar intra-order match scores in both the $r$-th and $l$-th order, it will be considered as more valuable and reliable, and thus will have larger inter-level potentials.

V-B Optimization

The objective function in Eq. 22 properly integrates the information of multi-grained objects. Thus, by maximizing Eq. 22, we are able to obtain the optimal one-to-one mapping result among individuals in probe and gallery groups.

To solve the multi-order matching problem in Eq. 22, we construct a multi-order association graph to incorporate all candidate matches & multi-order potentials in the objective function, as in Fig. 5. In Fig. 5, each layer includes all candidate matches $c_{ij}$ (the circular nodes) and their corresponding intra-order matching scores $m_{k}$ (the rectangular nodes in green, orange, or pink), which models the intra-order potentials in a specific order. Besides, the blue rectangular nodes linking circular nodes in different layers represent the inter-order correlations $m_{rl}(c_{ij},r,l)$. They model the inter-order potentials between different orders.

With this association graph, we are able to solve Eq. 22 by adopting general-purpose hyper-graph matching solvers [38, 39]. Specifically, we first initialize a mapping probability for each candidate match in the associate graph, and then apply reweighted random walk [39] to update these mapping probabilities via the inter/intra-order links and potential weights in the association graph. Finally, the mapping probabilities in all layers in the association graph are combined to obtain the optimal one-to-one mapping result from the candidate matches [38].

V-C Fused Matching Score

After obtaining one-to-one mapping between individual people in two groups, we are able to calculate matching scores accordingly. In order to obtain a more reliable matching score, we introduce a fused scheme by integrating the information of both matched and unmatched objects:

[TABLE]

where $({i_{1},..,i_{k}})$ is a person/subgroup in probe group image $I_{p}$, $M({i_{1}..i_{k}})$ is its one-to-one matched person/subgroup in gallery image $I_{g}$. $w_{k}(\cdot)$ is the similarity matching score between $({i_{1},..,i_{k}})$ and $M({i_{1}..i_{k}})$, as in Eqs. 24 and 26. $\alpha$ is the importance weight. $\lambda_{r}$=$0.5$ is a balancing factor. $\mathcal{\mathcal{R}}_{p}$ and $\mathcal{\mathcal{R}}_{g}$ are the sets of reliably matched objects in groups $I_{p}$ and $I_{g}$, and $\overline{\mathcal{\mathcal{R}}}_{p}$ and $\overline{\mathcal{\mathcal{R}}}_{g}$ are the unmatched object sets. The matched object pairs that maximizing objective function 22 are taken as the reliably matched objects, and put into $\mathcal{\mathcal{R}}_{p}$ and $\mathcal{\mathcal{R}}_{g}$. The remaining unmatched or less similar objects are put into $\overline{\mathcal{\mathcal{R}}}_{p}$ and $\overline{\mathcal{\mathcal{R}}}_{g}$.

From Eq. V-C, our fused scheme integrates four granularities (i.e., $k=1,2,3,g$) to compute the group-wise matching score. Inside each granularity, the matched pairs that maximize Eq.  22 are used to compute the similarity (the first term in Eq. V-C), so as to reduce the interference of confusing or mismatched people/people subgroups. Meanwhile, we introduce an unmatched term evaluating the importance of unmatched objects (the second term in Eq. V-C). As such, we can properly avoid misleadingly high matching scores in false group pairs (as in Fig. 6a) and obtain a more reliable result.

VI Experimental Results

We perform experiments on three datasets: (1) the publicly available i-LID MCTS dataset [4] which contains $274$ group images for $64$ groups; (2) our newly constructed DukeMTMC Group dataset which includes $177$ group image pairs extracted from the $8$-camera-view DukeMTMC dataset [40]; (3) our newly collected Road Group dataset which consists of $162$ group pairs taken from a $2$-camera crowded road scene111Dataset and source code will be available at http://min.sjtu.edu.cn/lwydemo/GroupReID.html..

To construct the Road Group dataset, we use [41] to automatically identify groups from key frames that were extracted at equi-intervals of 50 frames. Then, the group image pairs are randomly selected from sets according to different group sizes and occlusion variations. We define two cross-view images from different cameras as belonging to the same group when they have more than $60\%$ members in common.

Some example of groups from the three datasets are shown in Fig. 3b, showing diverse challenging conditions across cameras. We also provide a statistical analysis of the three datasets in Table I. ‘Average Person’ denotes the average number of individuals per group image, while ‘Member Change’ denotes the average difference in group size for each pair of group images. ‘Dispersity’ is measured by averaging the normalized distance between each individual member to its group centroid. This measure computes the sparsity (or compactness) of the group, which indirectly indicates the proneness to layout change. ‘Occlusion Pairs’ denotes the average number of individual pairs that occlude each other within an image. Note that despite the i-LID MCTS dataset having smaller and more compact groups, it suffers from low image quality and large illumination changes. Meanwhile, the new datasets, DukeMTMC Group and Road Group are both plagued with severe object occlusions, and large layout and group member variations due to larger group sizes.

VI-A Experiment Setup

To provide a fair comparison with other methods, we follow the evaluation protocol in [8, 42] by partitioning the datasets by half into training set and validation set. The final results are obtained by averaging the performances on the validation set over $5$ random splits. We report our results using Cumulated Matching Characteristic (CMC) [3], which is able to measure rank-$k$ correct match rates.

We use a ResNet-34 model as our model feature extractor. We initialize the learning rate at $10^{-4}$ and divide it by a factor of $10$ every $15$ epochs, stopping at the maximum epoch of $40$. Our network is trained with a SGD solver (weight decay of $10^{-4}$) on an NVIDIA GeForce GTX 1080 GPU. Contrary to the work of [33], we employ a non-conventional bin distribution of size $6\times 3$ for ROI Pooling since bounding boxes for people are typically tall and narrow.

VI-B Ablation Studies

We present our results alongside a number of ablation studies to provide a wide perspective of the performance of our proposed approach together with the impact of various components in the framework.

VI-B1 Results with features of different granularities

In order to evaluate the effectiveness of our multi-grained group Re-ID framework, we compare eight methods of different granularities including some variations: (1) Only using global features [15] of the entire group (Global); (2) Only using features of individual people (Fine); (3) Using features of individual people and two-people subgroup (Fine+Medium); (4) Using features of individual, two-people, three-people subgroups (Fine+Medium+Coarse); (5) Using our multi-grained framework, but assigning equal importance weights for all people/people subgroups, i.e. set all to $1$ (Proposed-equal weights); (6) Using our multi-grained framework, but omitting the spatial relation features in the multi-grained representation (cf. Sec. III), Proposed-no spatial); (7) Using our multi-grained framework, but using the ground truth pedestrian detection results (Proposed-GT); (8) Using our multi-grained framework with automatic pedestrian detection method of [43] to identify individual people in groups (Proposed-auto).

Table II shows the CMC results of group Re-ID on the Road Group dataset, measuring the correct match rates for different Re-ID ranks. The upper part lists the results based on hand-crafted features (cf. Sec. III-A) while the lower part lists the results based on deep convolutional features (cf. Sec. III-B).

Fig. 7 shows some group-wise matching results under different methods. We make the following observations:

(1) The Global method achieves poor results. This implies that simply using the entire group image cannot effectively handle the intricate variations that are present in group Re-ID. Comparatively, the Fine method has obviously better performance by extracting and matching individual people to handle the challenges of group dynamics. However, its performance is still hindered by the interference of pedestrian misdetections or mismatches (cf. Fig. 7a). These problems are reduced more effectively in the Fine+Medium and Fine+Medium+Coarse methods, both of which, contain sub-group level information that captures underlying group dynamics. Our proposed framework (Proposed-GT and Proposed-auto), which includes all levels of granularity, can achieve the best performance.

(2) The Proposed-equal weights method has obviously poorer results than its counterpart with importance weights (Proposed-GT and Proposed-auto). This clearly indicates that: a) assigning importance weights to different individuals/people subgroups is significant in guaranteeing group Re-ID performances; b) Our proposed importance evaluation scheme is effective in finding proper importance weights for all levels of granularity, such that reliable and discriminative individuals/people subgroups are highlighted, resulting in better matching results. For instance, in Fig. 7c-7d, due to large layout change between group, the Proposed-equal weights scheme is unable to assign a high score on the pairs, while Proposed-auto scheme allows salient objects to be given greater importance, hence resulting in a higher matching score.

(3) The Proposed-no spatial method achieves relatively satisfactory results. This indicates that even when spatial relation features are not encoded, our approach can generally still obtain reliable performances, propelled by multi-grained information and importance weights. In the case of deep convolution features, we observe a relatively larger performance drop when spatial features are not used (about 10% for rank-1), which indicates that the choice of spatial features extracted from our CNN is crucial and can boost the group Re-ID accuracy to a large extent.

(4) The Proposed-auto method has almost similar results to the Proposed-GT method, only marginally lower in most cases. The close performances of these two methods indicate that our multi-grained group Re-ID framework has the ability to handle matching errors caused by pedestrian misdetections. For example, in Fig. 7a, the left group in camera $A$ is incorrectly matched with the blue rectangle in camera $B$ which detected a parked motorcycle. However, by integrating multi-grained information , we can successfully avoid this mismatch by considering subgroup correlation at higher-level granularities (cf. Fig. 7b). Note that the iterative procedure refines the importance weights even at the fine granularity, subsequently resulting in a correct individual match.

(5) By comparing the results on a whole, we find a similar trend with respect to the combination of granularities, which affirms the good scalability and extensibility of our group Re-ID framework to accommodate different types of features. Further, we find when deep convolutional features are used, the performance of our framework is usually better than the hand-crafted counterpart on identical granularity settings.

VI-B2 Results with different detection recalls

In Sec. VI-B1 we demonstrate that the matching accuracy is competitive when a high quality pedestrian detector is used. Following this observation, we further investigate the effect of detection recalls on our final matching accuracy. We conduct this experiment by altering the output confidence threshold of our detector [43] to obtain detection results at different recall rates. We then perform group Re-ID based on the detected objects and their respective bounding boxes In Fig. 8, we compare the Rank-1 CMC scores for hand-crafted and deep convolutional features against detection recall rates.

From Fig. 8, we find the final matching results is relatively robust against the quality of detectors that we adopt. This is evident as the drop in Rank-1 CMC score is less than $3\%$ for every corresponding $5\%$ decrease (approximately) in recall rate. This observation further demonstrates that our multi-grained matching framework is robust to the detection quality and could still be helpful for group Re-ID problem even when the prior knowledge of individuals is incomplete.

VI-B3 Results with different matching constraints

In this study, we further evaluate the effectiveness of our multi-order matching process when matching constraints are varied. Five methods are compared: (1) Only single level matching, i.e., the first-order potential in Eq. 23 (Single); (2) Discard the inter-order potential term i.e., setting all \mathcal{{\color[rgb]{0,0,0}P}}_{rl}(\mathcal{C}) terms in Eq. 22 to [math], (No inter); (3) Discard the unmatched term (i.e., the second term in Eq. V-C) when calculating matching scores (No dis); (4) Use hyper-edge matching [39] , which integrates multi-grained information by constructing a multi-order similarity function (Hyp-E); (5) Our proposed matching process (Proposed-auto).

Table IV shows the CMC matching scores for group Re-ID task using multi-order matching with different matching constraints and criterion. We report results for both hand-crafted features and convolutional features on the Road Group dataset. From Table IV, we can draw the following conclusions:

(1) In comparison with the Single method, the higher-order potentials of the Proposed-auto method clearly play an important role in improving group Re-ID score by a great measure. These potentials are essential to handle matching of multiple-person subgroups to complement the use of multi-grained object representation

(2) The Proposed-auto method obtained better results than the No inter method. This comparison indicates that the inter-order potential term is useful to properly capture correlations between different levels of granularity.

(3) The Proposed-auto method performed significantly better than the No dis method. This demonstrates the importance of including the information of unmatched objects (cf. Eq. V-C) in the matching process. Results on both features show that this information has far more impact than that of the inter-order potential term (cf. method (2)).

(4) The Proposed-auto method also has better matching accuracy than the Hyp-E method. This demonstrates that our multi-order matching process can make better use of the multi-grained information in groups during matching.

VI-B4 Results with different feature combination

We also investigate the effectiveness of using convolutional features from our multi-task CNN for different parts of the extracted features (see Fig. 3a). We evaluate the CMC score for all three datasets on the following combination of features: (1) Use hand-crafted features to describe both appearance and spatial relation for objects of all granulairity (Hand-crafted). (2) Use convolutional features only to represent the global image and keep other features as hand-crafted (Global-conv). (3) Represent appearance of global and local objects with convolutional features and keep spatial relation features as hand-crafted. (Appearnce-conv). (4) Convolutional features for all parts including the spatial relation features. (Full-conv).

We report the matching accuracy with respect to different ranks in Table III. From these results, we can observe that:

(1) Benefits are limited when only the global handcrafted feature is replaced with one of deep convolutional. This shows that convolutional features are not sufficiently discriminative when applied on the entire group image.

(2) There is a leap in improvement from Appreance-conv to Full-conv method, which indicates that using deep representations to encode spatial relations between individuals is more impactful than opting for hand-crafted representations.

(3) Overall, the improvement brought on by Full-conv method over the Hand-crafted method is more prominent on DukeMTMC Group and Road Group datasets than on i-LIDS MCTS. This is indicative of the robustness of our proposed deep convolutional features in more crowded scenarios. However, the handcrafted feature is still valuable since it can be applied on any scenarios without additional training process, especially when the data is limited and/or there are insufficient means to train a CNN feature extractor.

VI-C Results on Single Person Re-ID

Although our approach is designed for group Re-ID task, the intermediate result of fine-grained mapping $\mathcal{C}$ could be seen as a side product of our matching process. To further investigate how multi-order constraints and priors of group pairs effect the accuracy of individual matching, we conduct an extra experiment on single person Re-ID.

We compare two state-of-the-art single Re-ID methods, i.e. TriNet [12] and AlignReID [26], against four variants of our matching scheme on Road Group dataset: (1) Given a person from a probe image, we find the nearest person from among all individuals in the gallery groups based on Euclidean distance in feature space (Single-match). (2) Given a probe image, we first find its matched group in gallery, and for each individual in probe group image, we find the nearest person from among the individuals in the matched gallery group based on Euclidean distance in feature space (Intra-group). (3) We first obtain the matched pairs $\mathcal{C}$ between groups by solving the group-wise multi-order matching problem, if the matched group is exactly the ground truth, we take $\mathcal{C}$ as matching results for individual objects in probe image, otherwise we regard all persons in probe groups as unmatched. (Proposed-auto). We conduct these experiments on both hand-crafted and deep convolutional features.

We report the Rank-1 CMC score for single person Re-ID on the Road Group dataset in Table V and further visualize some sample results of different schemes in Fig. 9a and Fig. 10. Table V is split into two parts. The upper part lists results from methods without prior for groups, while the lower part lists results with group constraint. From these results, we observe the following:

(1) From Table V, we observe that simple person-wise matching strategy without prior of groups performs rather poorly compared with other approaches. This indicates person Re-ID using only person-wise descriptors may be ill-suited for such group scenarios since the search space is likely too large with limited samples per person. Without group priors, it is common to yield matched individuals from other groups who are similar in appearance (illustrated by the blue arrows in Fig. 9a-9b).

(2) Although Intra-group method performs better than person-wise matching, it is still worse than the proposed-auto scheme. Even by limiting the search space of the matched group, there still exists interferences that results in incorrect individual matching For example, in Fig. 10, the matched results in Fig. 10a show confusion with an individual with highly similar clothes; in Fig. 10b, there exists an occlusion in an overcrowded area. Both of these factors result in failures in intra-group method, while proposed-auto scheme could handle these issues better since the constrain of multi-order potential requires the system to consider optimal matching not only between individuals but also between sub-groups of multiple person.

VI-D Comparison with State-of-the-art Methods

Table VI summarizes the group Re-ID performances on various datasets, comparing our proposed approach against state-of-the-art group Re-ID methods: CRRRO-BRO [4], Covariance [5], PREF [7], BSC+CM [8]. For clarity, we denote the features used in our method using the suffix hand for hand-crafted features and conv for the convolutional features derived from our multi-task deep CNN.

For further benchmarking we also include the results of state-of-the-art methods designed for single person Re-ID. Among them are methods that utilize patch saliency (Saliency [6]) or a KMFA($R_{\chi^{2}}$) distance metric to calculate image-wise similarity (Mirror+KMFA [15]). We also compare with two deep metric learning based methods: TriNet[12], a combination of CNN and triplet loss and AlignReID [26], a CNN-based method which simultaneously learns global and local distances between sample images. Since these two methods are originally designed for person Re-ID, we design two variants to extend them for the group Re-ID scenario. One variant extracts features of individuals, under their respective deep frameworks [12, 26], and proceeds to apply Kuhn-Munkres’ algorithm for bipartite matching between individuals in two groups. Finally, the similarity between two groups is computed as the inverse of the summation of feature distances between matched pairs. We denote this variant with a suffix local, named after the nature of this method. The other variant directly takes the group image as the input of the algorithm in [12, 26] and calculates the group similarity according to Euclidean distance between output features. We denote this variant with a suffix global, since it considers the whole image. From Table VI, we can observe that:

(1) Our approach (handcrafted or deep convolutional features) has better results than the other competing methods, on all three evaluated datasets. This demonstrates the resounding consistency and effectiveness of our approach in addressing the group Re-ID problem.

(2) Group Re-ID methods that used global features (CRRRO-BRO [4], Covariance [5], PREF [7]) achieve less satisfactory results. This indicates that utilizing only global features is clearly inadequate at handling the diverse range of challenges in group Re-ID.

(3) Although the BSC+CM method obtained better results than that of global feature-based methods by introducing fine-grained objects (i.e. patches) to handle group dynamics, its performance is still evidently lower than our approaches. This implies the usefulness of including information from multiple granularities.

(4) Our approach is also obviously superior to the conventional methods for single person Re-ID (Saliency, Mirror+KMFA). This indicates that the task of re-identifying each individual in a group-wise setting is a rather limited solution that is likely to fail in challenging group Re-ID scenarios.

(5) Deep metric-learning methods perform poorer than our approach since they only resort to fine-grained representation (between person objects) while ignoring more complex patterns that occur at the medium and coarse sub-group levels. Interestingly, this comparison also shows that these single Re-ID methods perform relatively better on groups with more individuals (e.g. Road Group).

(6) The improvement of our approach is more obvious on datasets with larger group layouts and group member changes (DukeMTMC Group & Road Group). This demonstrates that our approach is capable of handling the dynamic changes that naturally occur in the group membership. On the other hand, the improvement on i-LDS MCTS dataset is less obvious. This is the result of limited volume of people in this dataset, since for such scenarios, representations based on multiple levels of granularity are less discriminative than when applied to crowded scenes; purely global descriptors appear to be sufficiently competitive for characterizing such small groups.

VI-E Computational Complexity

Table VII shows the running time of our group Re-ID approach on different datasets (excluding the time consumed for object detection and feature extraction). The running test is conducted on an 8-core i7-7700@3.60GHz CPU platform.

We list two time complexity values: (1) the running time for the entire process (All image pairs in the dataset), and (2) the average running time for computing the similarity of a single group image pair (Per image pair). Table VII shows that our approach is acceptable in running time.

VII Conclusion

This paper introduces a novel approach to address the seldom-studied problem of group re-identification. Our work contributes broadly in these aspects: 1) a multi-grained group Re-ID framework which derives feature representations for multi-grained objects and iteratively evaluates their importance at different granularities to handle group dynamics; 2) a multi-order matching process which integrates multi-grained information to obtain more reliable group matching results; 3) two independent pipelines (handcrafted and deep learning) which are capable of encoding appearance and spatial relations of multi-grained objects. Overall, our extensive experiments demonstrate the viability of our approaches. We also release our group Re-ID datasets involving realistic challenges to spur future works towards this direction.