An Efficient Framework for Visible-Infrared Cross Modality Person Re-Identification

TL;DR

This paper introduces a four-stream deep learning framework utilizing diverse image representations and local pattern maps to significantly enhance visible-infrared person re-identification performance, outperforming existing methods on benchmark datasets.

Contribution

The paper presents a novel multi-stream CNN architecture with local pattern maps and a re-ranking post-processing step for improved VI-ReId accuracy.

Findings

Achieved 29.79% Rank-1 improvement on SYSU-MM01 dataset.

Improved mAP by 30.91% on SYSU-MM01 dataset.

Outperformed state-of-the-art methods by large margins.

Abstract

Visible-infrared cross-modality person re-identification (VI-ReId) is an essential task for video surveillance in poorly illuminated or dark environments. Despite many recent studies on person re-identification in the visible domain (ReId), there are few studies dealing specifically with VI-ReId. Besides challenges that are common for both ReId and VI-ReId such as pose/illumination variations, background clutter and occlusion, VI-ReId has additional challenges as color information is not available in infrared images. As a result, the performance of VI-ReId systems is typically lower than that of ReId systems. In this work, we propose a four-stream framework to improve VI-ReId performance. We train a separate deep convolutional neural network in each stream using different representations of input images. We expect that different and complementary features can be learned from each stream. In our framework, grayscale and infrared input images are used to train the ResNet in the first stream. In the second stream, RGB and three-channel infrared images (created by repeating the infrared channel) are used. In the remaining two streams, we use local pattern maps as input images. These maps are generated utilizing local Zernike moments transformation. Local pattern maps are obtained from grayscale and infrared images in the third stream and from RGB and three-channel infrared images in the last stream. We improve the performance of the proposed framework by employing a re-ranking algorithm for post-processing. Our results indicate that the proposed framework outperforms current state-of-the-art with a large margin by improving Rank-1/mAP by 29.79%/30.91% on SYSU-MM01 dataset, and by 9.73%/16.36% on RegDB dataset.

Tables

Table 1. Table 1 : ResNet architectures used in this study. The building blocks are given in brackets. Each row in the brackets indicates the kernel sizes and the number of output channels of convolution layers used in the building blocks.

50-layer	101-layer	152-layer
7 x 7, 64, stride 2
3 x 3 max pool, stride 2
[ $1\times 1,64$ $3\times 3,64$ $1\times 1,256$ ] x 3	[ $1\times 1,64$ $3\times 3,64$ $1\times 1,256$ ] x 3	[ $1\times 1,64$ $3\times 3,64$ $1\times 1,256$ ] x 3
[ $1\times 1,128$ $3\times 3,128$ $1\times 1,512$ ] x 4	[ $1\times 1,128$ $3\times 3,128$ $1\times 1,512$ ] x 4	[ $1\times 1,128$ $3\times 3,128$ $1\times 1,512$ ] x 8
[ $1\times 1,256$ $3\times 3,256$ $1\times 1,1024$ ] x 6	[ $1\times 1,256$ $3\times 3,256$ $1\times 1,1024$ ] x 23	[ $1\times 1,256$ $3\times 3,256$ $1\times 1,1024$ ] x 36
[ $1\times 1,512$ $3\times 3,512$ $1\times 1,2048$ ] x 3	[ $1\times 1,512$ $3\times 3,512$ $1\times 1,2048$ ] x 3	[ $1\times 1,512$ $3\times 3,512$ $1\times 1,2048$ ] x 3
global average pool

Table 2. Table 2 : Results on SYSU-MM01 dataset obtained with different values of n 𝑛 n and k 𝑘 k parameters of the LZM transformation.

Method	n	k	Rank1	Rank10	Rank20	mAP
	2	5	30.77	78.74	89.67	32.28
	3	3	29.18	76.14	88.25	30.79
$R{50}_{LZM\left(gray\_ir\right)}$	3	5	31.45	79.27	90.48	32.57
	3	7	31.48	78.05	89.29	32.64
	4	5	30.35	78.79	89.93	31.92
	5	5	29.23	75.97	88.39	31.06

Table 3. Table 3 : Results on SYSU-MM01 dataset obtained when ResNet-50 initialized with scaled Gaussian distribution [ 53 ] is used in all the streams of the framework.

Method	Rank1	Rank10	Rank20	mAP
$R{50}_{gray\_ir}$	28.59	75.95	87.57	30.42
$R{50}_{rgb\_ir}$	26.20	70.53	83.24	28.48
$R{50}_{gray\_ir}+R{50}_{rgb\_ir}$	33.06	79.67	89.87	35.11
$R{50}_{LZM\left(gray\_ir\right)}$	31.45	79.27	90.48	32.57
$R{50}_{LZM\left(rgb\_ir\right)}$	28.32	73.09	84.98	30.44
$R{50}_{LZM\left(gray\_ir\right)}+R{50}_{LZM\left(rgb\_ir\right)}$	34.73	81.13	91.26	36.30
$All$	38.39	83.75	92.47	39.67

Table 4. Table 4 : Results on SYSU-MM01 dataset obtained when pre-trained ResNet-50 is used in all the streams of the framework.

Method	Rank1	Rank10	Rank20	mAP
$R{50}_{gray\_ir}$	38.94	85.47	93.99	39.63
$R{50}_{rgb\_ir}$	33.87	76.99	88.66	35.21
$R{50}_{gray\_ir}+R{50}_{rgb\_ir}$	42.58	86.61	94.54	43.38
$R{50}_{LZM\left(gray\_ir\right)}$	37.30	84.56	93.60	37.93
$R{50}_{LZM\left(rgb\_ir\right)}$	35.09	80.11	91.18	36.67
$R{50}_{LZM\left(gray\_ir\right)}+R{50}_{LZM\left(rgb\_ir\right)}$	41.50	86.44	94.73	42.33
$All$	45.00	89.06	95.77	45.94

Table 5. Table 5 : Results on SYSU-MM01 dataset obtained when pre-trained ResNet-101 is used in all the streams of the framework.

Method	Rank1	Rank10	Rank20	mAP
$R{101}_{gray\_ir}$	40.41	85.02	93.80	41.21
$R{101}_{rgb\_ir}$	35.71	79.34	89.69	38.01
$R{101}_{gray\_ir}+R{101}_{rgb\_ir}$	43.70	86.89	94.71	45.35
$R{101}_{LZM\left(gray\_ir\right)}$	39.98	86.39	94.88	40.94
$R{101}_{LZM\left(rgb\_ir\right)}$	37.41	81.92	92.17	39.25
$R{101}_{LZM\left(gray\_ir\right)}+R{101}_{LZM\left(rgb\_ir\right)}$	43.58	88.35	96.02	44.94
$All$	46.80	89.99	96.62	48.21

Table 6. Table 6 : Results on SYSU-MM01 dataset obtained when pre-trained ResNet-152 is used in all the streams of the framework.

Method	Rank1	Rank10	Rank20	mAP
$R{152}_{gray\_ir}$	38.88	85.01	93.80	40.23
$R{152}_{rgb\_ir}$	36.27	78.63	88.74	38.33
$R{152}_{gray\_ir}+R{152}_{rgb\_ir}$	43.54	86.49	94.40	45.30
$R{152}_{LZM\left(gray\_ir\right)}$	40.68	84.89	93.32	41.02
$R{152}_{LZM\left(rgb\_ir\right)}$	39.65	81.69	91.36	40.49
$R{152}_{LZM\left(gray\_ir\right)}+R{152}_{LZM\left(rgb\_ir\right)}$	44.69	87.00	94.71	45.21
$All$	47.35	89.10	95.67	48.32

Table 7. Table 7 : Results on SYSU-MM01 dataset obtained using ResNet-50, ResNet-101 and ResNet-152 models in different combinations. R ​ X { g + r } 𝑅 subscript 𝑋 𝑔 𝑟 RX_{\{g+r\}} and R ​ X { L ​ Z ​ M ​ ( g + r ) } 𝑅 subscript 𝑋 𝐿 𝑍 𝑀 𝑔 𝑟 RX_{\{LZM(g+r)\}} represent the concatenation of the features from the first and second, and third and fourth streams, respectively.

Method	Rank1	Rank10	Rank20	mAP
$R{50}_{\left\{g+r\right\}}+R{50}_{\left\{LZM\left(g+r\right)\right\}}$	45.00	89.06	95.77	45.94
$R{50}_{\left\{g+r\right\}}+R{101}_{\left\{g+r\right\}}$	45.80	88.51	95.53	46.94
$R{50}_{\left\{g+r\right\}}+R{101}_{\left\{LZM\left(g+r\right)\right\}}$	46.69	90.32	96.59	47.79
$R{50}_{\left\{g+r\right\}}+R{152}_{\left\{g+r\right\}}$	46.02	88.25	95.44	47.17
$R{50}_{\left\{g+r\right\}}+R{152}_{\left\{LZM\left(g+r\right)\right\}}$	47.51	89.59	96.01	48.15
$R{101}_{\left\{g+r\right\}}+R{50}_{\left\{g+r\right\}}$	45.80	88.51	95.53	46.94
$R{101}_{\left\{g+r\right\}}+R{50}_{\left\{LZM\left(g+r\right)\right\}}$	46.23	89.22	96.11	47.42
$R{101}_{\left\{g+r\right\}}+R{101}_{\left\{LZM\left(g+r\right)\right\}}$	46.80	89.99	96.62	48.21
$R{101}_{\left\{g+r\right\}}+R{152}_{\left\{g+r\right\}}$	45.84	88.13	95.34	47.50
$R{101}_{\left\{g+r\right\}}+R{152}_{\left\{LZM\left(g+r\right)\right\}}$	47.24	89.12	96.09	48.27
$R{152}_{\left\{g+r\right\}}+R{50}_{\left\{g+r\right\}}$	46.02	88.25	95.44	47.17
$R{152}_{\left\{g+r\right\}}+R{50}_{\left\{LZM\left(g+r\right)\right\}}$	46.57	89.16	95.97	47.75
$R{152}_{\left\{g+r\right\}}+R{101}_{\left\{g+r\right\}}$	45.84	88.13	95.34	47.50
$R{152}_{\left\{g+r\right\}}+R{101}_{\left\{LZM\left(g+r\right)\right\}}$	47.01	89.63	96.33	48.37
$R{152}_{\left\{g+r\right\}}+R{152}_{\left\{LZM\left(g+r\right)\right\}}$	47.35	89.10	95.67	48.32

Table 8. Table 8 : State-of-the-art comparison on SYSU-MM01 dataset under all-search mode.

Method	Rank1	Rank10	Rank20	mAP
Lin et al.* [56]	5.29	33.71	52.95	8.00
One-stream [1]	12.04	49.68	66.74	13.67
Two-stream [1]	11.65	47.99	65.50	12.85
Zero-padding [1]	14.80	54.12	71.33	15.95
TONE+XQDA* [3]	14.01	52.78	69.06	15.97
TONE+HCML* [3]	14.32	53.16	69.17	16.16
BCTR [2]	16.12	54.90	71.47	19.15
BDTR [2]	17.01	55.43	71.96	19.66
Kang et al. [5]	23.18	51.21	61.73	22.49
cmGAN [13]	26.97	67.51	80.56	27.80
eBDTR(ResNet50) [57]	27.82	67.34	81.34	28.42
D2RL [4]	28.90	70.60	82.40	29.20
Ye et al. [58]	31.41	73.75	86.29	33.18
MAC [59]	33.26	79.04	90.09	36.22
$R{50}_{\left\{g+r\right\}}+R{50}_{\left\{LZM\left(g+r\right)\right\}}$	45.00	89.06	95.77	45.94
$R{101}_{\left\{g+r\right\}}+R{101}_{\left\{LZM\left(g+r\right)\right\}}$	46.80	89.99	96.62	48.21
$R{152}_{\left\{g+r\right\}}+R{152}_{\left\{LZM\left(g+r\right)\right\}}$	47.35	89.10	95.67	48.32
All	48.87	90.73	96.72	49.85
All + re-ranking	63.05	93.62	96.30	67.13
* indicates the results copied from [2].

Table 9. Table 9 : State-of-the-art comparison on SYSU-MM01 dataset under indoor-search mode.

Method	Rank1	Rank10	Rank20	mAP
Lin et al.* [56]	9.46	48.98	72.06	15.57
One-stream [1]	16.94	63.55	82.10	22.95
Two-stream [1]	15.60	61.18	81.02	21.49
Zero-padding [1]	20.58	68.38	85.79	26.92
cmGAN [13]	31.63	77.23	89.18	42.19
eBDTR(ResNet50) [57]	32.46	77.42	89.62	42.46
MAC [59]	33.37	82.49	93.69	44.95
Ye et al. [58]	37.62	83.27	93.56	46.32
$R{50}_{\left\{g+r\right\}}+R{50}_{\left\{LZM\left(g+r\right)\right\}}$	49.66	92.47	97.15	59.81
$R{101}_{\left\{g+r\right\}}+R{101}_{\left\{LZM\left(g+r\right)\right\}}$	53.01	94.05	98.44	62.86
$R{152}_{\left\{g+r\right\}}+R{152}_{\left\{LZM\left(g+r\right)\right\}}$	52.10	93.69	98.06	62.33
All	54.28	94.22	98.22	63.92
All + re-ranking	69.06	96.30	97.16	76.95
* indicates the results copied from [1].

Table 10. Table 10 : Results on RegDB dataset under visible to thermal setting.

Method	Rank1	Rank10	Rank20	mAP
$R{50}_{gray\_ir}$	43.94	67.28	77.50	45.28
$R{50}_{rgb\_ir}$	49.48	69.60	79.30	50.42
$R{50}_{gray\_ir}+R{50}_{rgb\_ir}$	55.12	75.37	83.85	56.22
$R{50}_{LZM\left(gray\_ir\right)}$	38.78	61.15	71.54	40.14
$R{50}_{LZM\left(rgb\_ir\right)}$	36.85	59.25	70.39	38.62
$R{50}_{LZM\left(gray\_ir\right)}+R{50}_{LZM\left(rgb\_ir\right)}$	44.11	65.94	75.84	45.70
$All$	57.03	76.10	84.34	58.06

Table 11. Table 11 : Results on RegDB dataset under thermal to visible setting.

Method	Rank1	Rank10	Rank20	mAP
$R{50}_{gray\_ir}$	45.62	69.13	78.80	45.51
$R{50}_{rgb\_ir}$	48.36	68.57	78.53	49.19
$R{50}_{gray\_ir}+R{50}_{rgb\_ir}$	54.85	74.05	82.91	55.34
$R{50}_{LZM\left(gray\_ir\right)}$	39.54	63.99	74.08	40.10
$R{50}_{LZM\left(rgb\_ir\right)}$	37.51	61.61	71.75	38.13
$R{50}_{LZM\left(gray\_ir\right)}+R{50}_{LZM\left(rgb\_ir\right)}$	45.18	67.50	77.57	45.53
$All$	57.17	76.62	84.88	57.56

Table 12. Table 12 : State-of-the-art comparison on RegDB dataset.

Method	Rank1	Rank10	Rank20	mAP
Lin et al.* [56]	17.28	34.47	45.26	15.06
One-stream* [1]	13.11	32.98	42.51	14.02
Two-stream* [1]	12.43	30.36	40.96	13.42
Zero-padding* [1]	17.75	34.21	44.35	18.90
TONE+XQDA* [3]	21.94	45.05	55.73	21.80
TONE+HCML [3]	24.44	47.53	56.78	20.80
BCTR [2]	32.67	57.64	66.58	30.99
BDTR [2]	33.47	58.42	67.52	31.83
eBDTR (AlexNet) [57]	34.62	58.96	68.72	33.46
Ye et al. [58]	35.42	53.75	64.08	36.42
MAC [59]	36.43	62.36	71.63	37.03
D2RL [4]	43.40	66.10	76.30	44.10
D-HSME [60]	50.85	73.36	81.66	47.00
$R{50}_{\left\{g+r\right\}}+R{50}_{\left\{LZM\left(g+r\right)\right\}}$	57.03	76.10	84.34	58.06
+ re-ranking	60.58	67.71	77.13	63.36
* indicates the results copied from [2].

Table 13. Table 13 : Results on Market-1501 dataset.

Method	Rank1	Rank10	Rank20	mAP
$R{50}_{gray}$	69.09	90.32	94.06	44.71
$R{50}_{rgb}$	85.18	96.05	97.33	68.81
$R{50}_{gray}+R{50}_{rgb}$	88.57	96.85	98.28	72.91
$R{50}_{LZM\left(gray\right)}$	60.96	87.38	91.30	35.86
$R{50}_{LZM\left(rgb\right)}$	76.31	92.22	95.31	53.36
$R{50}_{LZM\left(gray\right)}+R{50}_{LZM\left(rgb\right)}$	80.52	94.36	96.08	58.03
$All$	88.78	96.85	97.83	71.83

Equations

$$V_{nm}(\rho ,\theta )=R_{nm}(\rho )e^{-\hat{j}m\theta },\hat{j}=\sqrt{-1}$$

$${\rho }_{ij}=\sqrt{x_i^2+y_j^2},$$

$${\theta }_{ij}=ta{n}^{-1}(y_j/x_i).$$

$$R_{nm}(\rho )=\sum _{k=0}^{\frac{n-|m|}{2}}(-1{)}^k\frac{(n-k)!}{k!(\frac{n+|m|-2k}{2})!(\frac{(n-|m|-2k)}{2})!}{\rho }^{n-2k}.$$

$$Z_{nm}=\frac{2(n+1)}{\pi (N-1{)}^2}\sum _{i=0}^{N-1}\sum _{j=0}^{N-1}{V}_{nm}({\rho }_{ij},{\theta }_{ij})f(i,j),$$

$$Z_{nm}^k(i,j)=\frac{2(n+1)}{\pi (k-1{)}^2}\sum _{p,q=-\frac{k-1}{2}}^{\frac{k-1}{2}}{V}_{nm}^k(p,q)f(i-p,j-q)$$

$$V_{nm}^k(i,j)=V_{nm}({\rho }_{ij},{\theta }_{ij}).$$

$$K(n)=\bigg{\{}\begin{matrix}\frac{n(n+2)}{4}\text{\quad if $n$ is even},\\ \frac{(n+1)^{2}}{4}\text{\quad if $n$ is odd.}\end{matrix}$$

$$I_{LZM(gray\_ir)}=T_{LZM}(I_{gray/ir})$$

$$T_{LZM}(I)=[r{e}_{11}^k(I),i{m}_{11}^k(I),r{e}_{22}^k(I),i{m}_{22}^k(I),\cdots ,r{e}_{nm}^k(I),i{m}_{nm}^k(I)].$$

$$I_{LZM(rgb\_ir)}=[T_{LZM}(I_{rgb}^r),T_{LZM}(I_{rgb}^g),T_{LZM}(I_{rgb}^b)].$$

$$I_{LZM(rgb\_ir)}=[T_{LZM}(I_{ir}),T_{LZM}(I_{ir}),T_{LZM}(I_{ir})].$$

$$ECN(p,g_i)=\frac{1}{2M}\sum _{j=1}^{M}d(p{N}_j,g_i)+d(g_i{N}_j,p).$$

$$N(p,M)\leftarrow \{N(p,t),N(t,q)\}$$

$$N(p,t)=\{g_i^o|i=1,2,\cdots ,t\}$$

$$N(t,q)=\{N(g_i^o,q),\cdots ,N(g_t^o,q)\}$$

$$R({\mathcal{L}}_i,{\mathcal{L}}_j)=\sum _{b=1}^B[K+1-po{s}_i(b){]}_{+}\times [K+1-po{s}_j(b){]}_{+}.$$

$$lr\_new=lr\_init\times (1-\frac{iter}{max\_iter}{)}^{0.9}.$$

Full text

††journal: Journal of LaTeX Templates

1 Introduction

Person re-identification (ReId) can be defined as the retrieval of the images of a person from a gallery set, where the gallery and query sets consist of the images captured by different cameras with different field-of-views. It is an essential problem for various real-world scenarios, especially for security. Therefore, this problem has recently attracted considerable attention of researchers working in the fields of computer vision and machine learning.

The illumination conditions, visible body parts, occlusion, and background complexity can vary extremely in images captured by different cameras. Due to the dynamic nature of the scene, these conditions can also change in images recorded by the same camera. These challenges and typically low-resolution images make ReId a challenging computer vision problem. In the literature on this subject, many different methods have been proposed in the last decade that address ReId from various aspects. The vast majority of the methods have been developed considering only the images captured by visible cameras. However, in dark or poorly illuminated environments, visible cameras cannot capture features that can distinguish people.

Most of the surveillance cameras used at night or in the dark usually operate in infrared mode in order to cope with poor illumination. Therefore, matching the person images captured by visible and infrared cameras is an important issue for video surveillance or miscellaneous applications. This issue is studied in literature as visible-infrared cross-modality person re-identification (VI-ReId) [1, 2, 3, 4, 5]. VI-ReId is the problem of retrieving the images of a person from a gallery set consisting of RGB (or infrared) images, given an infrared (or RGB) query image. For ReId, one of the most important cues in person images is obtained from the color. Therefore, the lack of color information in infrared images makes VI-ReId a very challenging problem.

In this paper, we show that ResNet [6] architectures trained with the use of RGB and infrared images together can outperform the current state-of-the-art. These architectures are widely used for image classification and other computer vision problems. They can learn the common feature representations for RGB and infrared images of the same individual as well as the distinctive properties between the individuals better than the existing methods proposed for VI-ReId. In this study, we introduce a four-stream framework built with ResNet architectures. There is no weight sharing between the ResNets in the framework, and in order to obtain different and complementary features as much as possible, we train each of them using a different representation of input images. In the first stream, RGB images converted to grayscale and infrared images are used together. In this way, features are extracted using only the shape and pattern information. In the second stream, to take advantage of the color information, RGB images and the three-channel infrared images (obtained by repeating the infrared image) are used. As mentioned above, one of the most important cues for ReId is obtained from the color information. However, due to the lack of color in infrared images, local shape and pattern information is critical for VI-ReId. Therefore, in addition to the features obtained from the raw (RGB, grayscale and infrared) images in the first two streams of the framework, in the last two streams, features are derived from the local pattern maps. In the third stream, we first apply the local Zernike moments (LZM) transformation [7] on grayscale and infrared images to expose the local patterns in the images. With the LZM transformation, different numbers of local pattern maps are generated by computing the Zernike moments around each pixel. We train the ResNet in the third stream by using the LZM pattern maps of grayscale and infrared images. In the last stream of the framework, we generate the pattern maps by exposing the local patterns separately in R, G, and B channels.

The contributions of this paper are summarized as follows:

We propose a novel framework that consists of four streams. In order to obtain complementary features from each stream, we train a ResNet architecture in each stream by using a different representation of input images.

• 2.

This work is the first study employing the LZM transformation for ReId and training a deep convolutional neural network with the LZM pattern maps.

• 3.

Our framework111The project webpage: https://github.com/emrahbasaran/cross-re-id outperforms current state-of-the-art with a large margin by improving Rank-1/mAP by $29.79\%/$ $30.91\%$ on SYSU-MM01 dataset, and by $9.73\%/16.36\%$ on RegDB dataset.

2 Related Work

In recent years, deep convolutional neural networks (DCNN) have led to significant progress in many different computer vision areas. Person re-identification (ReId) is one of the challenging computer vision problems. In order to cope with issues such as illumination conditions, variations in pose, closure, and background clutter, researchers have developed many DCNN frameworks tackling ReId in different ways [8, 9, 10, 11, 12].

In the majority of the studies on ReId, only the images in the visible spectrum (RGB-based) are taken into account when addressing the challenges, and the developed methods are intended for RGB-based images only. There are only a few studies for visible-infrared or visible-thermal cross-modality person re-identification. However, they are important issues for video surveillance or miscellaneous applications that have to be performed in poorly illuminated or dark environments. In [1], the authors have analyzed different network structures for VI-ReId, including one-stream and two-stream architectures and asymmetric fully-connected layer. They have found the performance of the one-stream architecture to be better. In addition, they have proposed deep zero padding to contribute to the performance of the one-stream network. Since the authors train the architectures using grayscale input images, they do not take advantage of the color of RGB images. Kang et al. [5] propose a one-stream model for cross-modality re-identification. They create single input images by placing visible and infrared images in different channels or by concatenating them. Using these input images consisting of positive and negative pairs, they train a DCNN employing two-class classification loss. In the study, some pre-processing methods are also analyzed. Ye et al. [2] use a two-stream DCNN to obtain modality-specific features from the images. One stream of the network is fed with RGB while the other is fed with infrared (or thermal) images, and on top of the streams, there is a shared fully connected layer in order to learn a common embedding space. The authors train the two-stream model using multi-class classification loss along with a ranking loss proposed by them. A very similar two-stream model is employed in [3], but this model is trained utilizing contrastive loss instead of ranking loss used in [2]. To improve the discrimination ability of the features extracted by the two-stream model, they propose a hierarchical cross-modality metric learning. In this method, modality-specific and modality-shared metrics are jointly learned in order to minimize cross-modality inconsistency. In [13], a generative adversarial network (GAN) is proposed for visible-infrared re-identification. The authors use a DCNN, which is trained employing both multi-class classification and triplet losses, as a generator to extract features from RGB and infrared images. The discriminator is constructed as a modality classifier to distinguish between RGB and infrared representations. Another work utilizing a variant of GAN has introduced by Kniaz et al. [14] for visible-thermal re-identification. In their method, a set of synthetic thermal images are generated for an RGB probe image by employing a GAN framework. Then, the similarities between the synthetic probe and the thermal gallery images are calculated. Wang et al. [4] propose a network dealing with the modality discrepancy and appearance discrepancy separately. They first create a unified multi-spectral representation by projecting the visible and infrared images to a unified space. Then, they train a DCNN using the multi-spectral representations by employing triplet and multi-class classification losses. There are also some other works [8, 15] trying to improve RGB-based ReId by fusing the information from different modalities.

In the literature, most of the existing studies on cross-modality matching or retrieval have been performed for heterogeneous face recognition (HFR) and text-to-image (or image-to-text) matching in the past decade. Studies on HFR, which are more relevant to VI-ReId, can be reviewed in three perspectives [16]: image synthesis, latent subspace, and domain-invariant features. In the first group, before the recognition, face images are transformed into the same domain [17, 18, 19]. The approach followed by the studies in the second group is the projection of the data of different domains into a common latent space [20, 21, 22]. In the recent works exploring domain-invariant feature representations, deep learning based methods are proposed. In [23], the features are obtained from a DCNN pre-trained on visible spectrum images, and metric learning methods are used to overcome inconsistencies between the modalities. He et al. [24] build a model in which the low-level layers are shared, and the high-level layers are divided into infrared, visible, and shared infrared-visible branches. Peng et al. [25] propose to generate features from facial patches and utilize a novel cross-modality enumeration loss while training the network. In [26], Mutual Component Analysis (MCA) [27] is integrated as a fully-connected layer into DCNN, and an MCA loss is proposed.

In this study, we propose a framework that utilizes the pattern maps generated by local Zernike moments (LZM) transformation [7]. As robust and holistic image descriptors, Zernike moments (ZM) [28] are commonly used in many different computer vision problems, i.e., character [29], fingerprint [30] and iris [31] recognition, and object detection [32]. LZM transformation is proposed by Sariyanidi et al. [7] to utilize ZM at the local scale for shape/texture analysis. In this transformation, images are encoded by calculating the ZM around each pixel. Thus, the local patterns in the images are exposed, and a rich representation is obtained. In literature, LZM transformation is used in various studies such as face recognition [7, 33, 34, 35], facial expression [36, 37] and facial affect recognition [38], traffic sign classification [39], loop closure detection [40, 41], and interest point detector [42]. In these works, the features are obtained directly from the LZM pattern maps. Unlike them, in this study, we use the LZM pattern maps as input images to train DCNNs. This is the first study training DCNNs with the LZM pattern maps. However, there have been some attempts employing ZM in convolutional networks. Mahesh et al. [43] initialize the trainable kernel coefficients by utilizing ZM with different moment orders. In [44] and [45], learning-free architectures are built constructing the convolutional layers by using ZM. Sun et al. [46] introduce a novel concept of Zernike convolution to extend convolutional neural networks for 2D-manifold domains.

3 Proposed Method

In this section, we begin by giving the details of ResNet [6] architectures. Then, we describe the LZM transformation [7] and introduce our four-stream framework proposed for VI-ReId. Finally, we explain the ECN re-ranking algorithm [10] used as post-processing.

3.1 ResNet Architecture

We show the performance of our proposed framework by using three different ResNet architectures which have different depths. These architectures have 50, 101, and 152 layers (we call these architectures as ResNet-50, ResNet-101, and ResNet-152, respectively, in the rest of the paper), and their details are provided in Table 1.

Each ResNet model shown in Table 1 starts with a $7\times 7$ convolution layer followed by a $3\times 3$ max pooling. The next layers of the models consist of a different number of stacked building blocks with a global average pooling layer at their top. In Table 1, we show the building blocks in brackets where each row in the brackets indicates the kernel sizes and the number of output channels of convolution layers used in the building blocks.

3.2 LZM transformation

Zernike moments (ZM) [28] are commonly used to generate holistic image descriptors for various computer vision tasks [29, 30, 31]. Independent holistic characteristics of the images are exposed with the calculation of ZM of different orders [47]. In order to reveal the local shape and texture information from the images by utilizing ZM, local Zernike moments (LZM) transformation is proposed in [7].

Zernike moments of an image are calculated using orthogonal Zernike polynomials [28]. These polynomials are defined in polar coordinates within a unit circle and represented as follows:

[TABLE]

where $R_{nm}(\rho)$ are real-valued radial polynomials, and $\rho$ and $\theta$ are the radial coordinates calculated as

[TABLE]

[TABLE]

Here, $x_{i}$ and $y_{j}$ are the pixel coordinates scaled to the range of $[-1,1]$. $R_{nm}(\rho)$ polynomials are defined as

[TABLE]

In (1) and (4), $n$ and $m$ are the moment order and the repetition, respectively. They take values such that $n-m$ is even, $0\leq n$ and $0\leq m\leq n$. Real and imaginary components of some $V_{nm}(\rho,\theta)$ Zernike polynomials are shown in Figure 1. Finally, ZM of an image $f(i,j)$ are calculated as:

[TABLE]

In the LZM transformation, images are encoded in $k\times k$ neighborhood of each pixel, as follows:

[TABLE]

where ${V}_{nm}^{k}(p,q)$ represents the Zernike polynomials used as $k\times k$ filtering kernels and defined as

[TABLE]

We give real and imaginary components of some ${V}_{nm}^{k}(i,j)$ filters in Figure 1. In Figure 2, we show a few pattern maps generated using the LZM filters. The filters constructed using $m=0$ are not used in the LZM transformation since the imaginary components of Zernike polynomials are not generated. According to the moment order $n$, the number of complex filters is calculated using

[TABLE]

Since we create a pattern map with each of the real and imaginary components of complex filters, the total number of local pattern maps becomes $2\times K(n)$.

3.3 Person Re-Identification Framework

The proposed framework for VI-ReId is shown in Figure 3. This framework includes four streams, and in each stream, a separate ResNet architecture is trained using both infrared and RGB images. Since each of the ResNets accepts a different representation of input images, different and complementary features are obtained from each stream.

There is a large visual difference between infrared and RGB images since infrared images have a single channel with invisible light information. This visual difference is reduced by converting RGB images to grayscale in the upper stream of the proposed framework, and the training is performed using infrared and grayscale images. In this way, $ResNet_{gray\_ir}$ in the upper stream of the framework learns to extract robust features for the person images by utilizing only the shape and texture information.

For ReId, one of the most important cues for person images is obtained from the color. Therefore, in the second stream of the proposed framework, $ResNet_{rgb\_ir}$ is trained using RGB and three-channel infrared (created by repeating the infrared channel) images. Unlike the one that is trained in the upper stream, the model trained in this stream uses color information of the RGB images and extracts discriminative features for the images in different modalities with visually large differences.

In VI-ReId, due to the lack of color in infrared images, the local shape and pattern information has great importance while matching the infrared and RGB images. Therefore, in addition to the two streams mentioned above, there are two other streams in the proposed framework in order to obtain stronger features related to local shape and pattern information. In the third stream, first, the local patterns are exposed by applying the LZM transformation on the grayscale and infrared images. Then, $ResNet_{LZM(gray\_ir)}$ is trained using the generated LZM pattern maps. As indicated in Section 3.2, a number of complex pattern maps are obtained as a result of the LZM transformation. The input tensor $I_{LZM(gray\_ir)}$ for $ResNet_{LZM(gray\_ir)}$ is prepared by concatenating the real and imaginary components of the complex maps. $I_{LZM(gray\_ir)}$ is denoted as

[TABLE]

where $I_{gray/ir}$ represents the grayscale or infrared image and $T_{LZM}(I)$ is defined as

[TABLE]

In Eq. (10), $re_{nm}^{k}(I)$ and $im_{nm}^{k}(I)$ are the real and imaginary components of a complex LZM pattern map, which is generated using the moment order $n$, the repetition $m$, and the filter size $k$. In the last stream of the proposed framework, LZM transformation is applied separately for the R, G and B channels, and the pattern maps obtained from these channels are concatenated to form the input tensor $I_{LZM(rgb\_ir)}$ for $ResNet_{LZM(rgb\_ir)}$, such as

[TABLE]

Here, $I_{rgb}^{r}$, $I_{rgb}^{g}$, and $I_{rgb}^{b}$ are the R, G, and B channels of the input image $I$. For infrared images, as mentioned earlier, the infrared channel is repeated for R, G, and B, such as

[TABLE]

In order to train the ResNet architectures in the proposed framework, multi-class classification with softmax cross-entropy loss is employed. Thus, in addition to learning to extract common features for the images in different modalities, the ResNet models also learn to extract features that express differences between individuals. As shown in Figure 3, the loss for each stream is calculated separately during the training. In the evaluation phase, the final representations for the person images are constructed by concatenating the feature vectors obtained from each stream and normalized using $\ell_{2}$-norm.

3.4 Re-Ranking

In recent studies [10, 48, 49], it has been shown that re-ranking techniques have a significant contribution to the performance of person re-identification. Therefore, in this study, we utilize Expanded Cross Neighborhood (ECN) re-ranking algorithm [10] as a post-processing element to further improve the performance. In this algorithm, the distance between a probe image $p$ and a gallery image $g_{i}$ from a gallery set $G$ with $B$ images $G=\{g_{i}|i=1,2,\cdots,B\}$ is defined as

[TABLE]

Here, $pN_{j}$ and $g_{i}N_{j}$ are the $jth$ neighbors in the expanded neighbor sets $N(p,M)$ and $N(g_{i},M)$ of the probe and $ith$ gallery images, respectively. $d(\cdot)$ represents the distance between the images, and $M$ is the total number of neighbors in a set. In order to construct the expanded neighborhood sets, first, an initial rank list $\mathcal{L}(p,G)=\{g^{o}_{1},\cdots,g^{o}_{B}\}$ in increasing order is created for each image by calculating the pairwise Euclidean distances between all images in the probe and gallery sets. Then, the expanded neighbor set $N(p,M)$ for a probe image $p$ is given as

[TABLE]

where the set $N(p,t)$ consists of the $t$ nearest neighbors of probe $p$, and the set $N(t,q)$ contains the $q$ nearest neighbors of each of the images in $N(p,t)$ such that:

[TABLE]

The expanded neighbor set $N(g_{i},M)$ for gallery image $g_{i}$ is obtained in the same way. In Equation 13, as suggested by the authors of [10], we compute the distance between the pairs using a list comparison similarity measure proposed in [50] and defined as

[TABLE]

In this equation, $[\cdot]_{+}=max(\cdot,0)$, $K$ is the number of nearest neighbors to be considered, and $pos_{i}(b)$ and $pos_{j}(b)$ show the position of image $b$ in the rank lists $\mathcal{L}_{i}$ and $\mathcal{L}_{j}$, respectively. The distance $d$ between the pairs is calculated as $d=1-R$ after scaling the range of the values of $R$ between [math] and $1$. For the parameters $t$, $q$, and $K$ of ECN re-ranking algorithm, we use the same setting given in [10] such that $t=3$, $q=8$, and $K=25$.

4 Experimental Results

4.1 Datasets

In this work, we evaluate the proposed framework on two different cross-modality re-identification datasets, SYSU-MM01 [1] and RegDB [15]. Additionally, we perform experiments on Market-1501 [51] to expose the performance of the framework for re-identification in the visible domain.

4.1.1 SYSU-MM01

The SYSU-MM01 [1] is a dataset collected for VI-ReId problem. The images in the dataset were obtained from $491$ different persons by recording them using four RGB and two infrared cameras. Within the dataset, the persons are divided into three fixed splits to create training, validation, and test sets. In the training set, there are $20284$ RGB and $9929$ infrared images of $296$ persons. The validation set contains $1974$ RGB and $1980$ infrared images of $99$ persons. The testing set consists of the images of $96$ persons where $3803$ infrared images are used as query and $301$ randomly selected RGB images are used as gallery. For the evaluation, there are two search modes, all-search and indoor-search. In the all-search mode, the gallery set consists of the images taken from RGB cameras (cam1, cam2, cam4, cam5), and the probe set consists of the images taken from infrared cameras (cam3 and cam6). In the indoor-search mode, the gallery set contains the RGB images only taken from the cameras (cam1 and cam2) located in the indoor. In the standard evaluation protocol of the dataset, the performance is reported using cumulative matching characteristic (CMC) curves and mean average precision (mAP). CMC and mAP are calculated with 10 random splits of the gallery and probe sets, and the average performance is reported. We follow the single-shot setting for all the experiments. In Figure 4, some sample images from SYSU-MM01 are shown.

4.1.2 RegDB

RegDB [15] consists of images captured using RGB and thermal dual-camera system. There are 412 persons in the dataset and each person has 20 images, 10 RGB and 10 thermal. We perform the experiments on this dataset by following the evaluation protocol proposed by Ye et al. [3]. In this protocol, the training and the test sets are created by randomly splitting the dataset into two parts. The gallery set is then created with thermal images, and the query set is created with RGB images. The results are reported with mAP and CMC, and the splitting process is repeated for 10 trials to ensure that the results are statistically stable. Sample images from RegDB are given in Figure 5.

4.1.3 Market-1501

Market-1501 dataset [51] has 32668 images captured from 1501 persons using one low-resolution and five high-resolution cameras. In the standard evaluation protocol of Market-1501, 12936 images of 751 persons are reserved for the training operations. The images of the remaining 750 persons constitute the query and gallery sets with 3368 and 19732 images, respectively. CMC and mAP metrics are used to report the performance on this dataset, as in SYSU-MM01 and RegDB.

4.2 Training the Networks

While training the ResNet networks in the proposed framework, we set the resolution of all RGB, infrared, and thermal input images to $492\times 164$. In the experiments where the LZM transformation is used, we normalize the images to have zero mean. For SYSU-MM01 and Market-1501, we perform the training operations using the own training set of each dataset and set the maximum number of iterations to 100K. However, for RegDB, we follow a two-step training strategy similar to [11] to avoid overfitting. Since the training set of RegDB has a limited number of images, models easily over-fit the training set. In the first step, the networks are trained for 100K iterations using a new set created by combining the training sets of SYSU-MM01 and RegDB. In the second step, the networks are fine-tuned for 10K iterations using only the training set of RegDB.

For all the training operations (including the fine-tuning on RegDB) performed in this work, we use Nesterov Accelerated Gradient [52] and set the mini-batch size to $8$, momentum to $0.9$ and weight decay to $0.0005$. We choose $0.01$ as the initial value of the learning rate and decay it 10 times using the exponential shift with the rate of $0.9$, such that

[TABLE]

Here, $lr\_init$ and $lr\_new$ are the initial and the updated learning rates, respectively. $iter$ is the current number of iterations and $max\_iter$ is the maximum number of iterations.

4.3 Re-Identification Performance on SYSU-MM01

In this section, we call the ResNet models in the streams of the proposed framework as $RX_{gray\_ir}$, $RX_{rgb\_ir}$, $RX_{LZM(gray\_ir)}$, and $RX_{LZM(rgb\_ir)}$, respectively, to avoid confusion while reporting the experimental results. The $X$ in the labels indicates which ResNet model is used and takes one of the values 50, 101, and 152. In this section, we show the experimental results under all-search mode.

As mentioned in Section 3.2, the number of filters used in the LZM transformation depends on the moment degree $n$ (Eq. 8). Therefore, we have first carried out experiments with model $R50_{LZM(gray\_ir)}$ in order to determine the optimal values for both the moment degree $n$ and the filter size $k$. The results obtained with different parameter sets are given in Table 2. According to this table, the results are close to each other, but slightly better results are obtained when $3/5$ and $3/7$ are used for the $n/k$ pair. For this reason, we prefer to use $n=3$ and $k=5$ to generate the LZM pattern maps for the rest of the study.

Table 3 shows the results obtained by training a ResNet-50 in each stream of the proposed framework. According to the results, better performance is achieved with $R50_{gray\_ir}$ and $R50_{LZM(gray\_ir)}$, where grayscale & infrared input images are used, compared to those of $R50_{rgb\_ir}$ and $R50_{LZM(rgb\_ir)}$ using RGB & infrared input images. Reducing the visual differences between infrared and RGB images by converting RGB images to grayscale makes a positive contribution to the performance. When the features obtained from the first two streams and the last two streams are combined, a significant performance increase is observed, as can be seen in the third and sixth rows of Table 3. With the concatenation of the features extracted by $R50_{gray\_ir}$ and $R50_{rgb\_ir}$, 4.5% improvement is achieved for Rank-1 as compared to $R50_{gray\_ir}$. Likewise, the concatenation of the features extracted by $R50_{LZM(gray\_ir)}$ and $R50_{LZM(rgb\_ir)}$ improves Rank1 by 3.3% as compared to $R50_{LZM(gray\_ir)}$. This shows that we generate complementary features with the models that are trained with grayscale & infrared and RGB & infrared input images. In the last row of Table 3, the results achieved by combining the features obtained in all streams of the proposed framework are given. When they are compared to those of $R50_{gray\_ir}+R50_{rgb\_ir}$ and $R50_{LZM(gray\_ir)}+R50_{LZM(rgb\_ir)}$, it is seen that the improvement obtained for Rank-1 is 5.3% and 3.7%, respectively. Therefore, taking this performance improvement into account, we can conclude that the models trained with the LZM pattern maps generate features that are different from and complementary to the ones produced by $R50_{gray\_ir}$ and $R50_{rgb\_ir}$.

While performing the experiments whose results are given in Table 2 and 3, we have initialized the weights of ResNet-50 with scaled Gaussian distribution proposed in [53]. However, many ReId studies [11, 9, 10] in the literature use pre-trained models instead of the models with randomly initialized weights. The vast majority of these pre-trained models are trained using ImageNet [54], which is a large-scale image classification dataset. In this way, the information of the model capable of image classification is utilized for ReId. By following the same approach, we have conducted experiments using ResNet-50 models, which are pre-trained on ImageNet, for each stream of our framework. The results are given in Table 4. In the third and fourth streams of the framework, the number of channels of the input images is greater than three because the LZM pattern maps are used. Therefore, in the experiments, we remove the first convolutional layer (conv1) of the pre-trained ResNet-50. Then, we insert a randomly initialized convolutional layer whose number of input channels matches the number of LZM pattern maps. For the first stream of the framework, we create three-channel input images by repeating the grayscale/infrared images for R, G, and B channels. If the results given in Tables 3 and 4 are compared, it is seen that we achieve a significant performance improvement for each stream by using pre-trained ResNet-50 models. With the concatenation of the features from four streams, we improve Rank-1 and mAP by 6.6% and 6.3% as compared to the results in the last row of Table 3 and reach to 45% and 45.9%, respectively.

According to recent research on computer vision tasks such as image classification, it is observed that deeper networks are more accurate [55]. In order to show the performance of our framework with deeper networks, we have conducted experiments using ResNet-101 and ResNet-152, which have more layers than ResNet-50. In the experiments, we have used the models pre-trained on ImageNet, and we show the results in Tables 5 and 6. As can be seen in the third and sixth rows of Tables 5 and 6, significant performance improvement is obtained with the concatenation of the features generated in the first two streams and in the last two streams, as in Tables 3 and 4. The highest results are achieved when the features from all the streams are combined. Using ResNet-101 and ResNet-152, we improve the best Rank1 result given in Table 4 (last row) by 1.8% and 2.4% and reach to 46.80% and 47.35%, respectively.

With the results given so far, we have demonstrated that the performance is significantly boosted by using the features learned from the LZM pattern maps. This shows that the models trained with the LZM pattern maps generate features that are different from and complementary to the ones generated by the other models. To further show the contribution of the features extracted from the LZM pattern maps, we have performed additional experiments and give the results in Table 7. In this table, $RX_{\{g+r\}}$ and $RX_{\{LZM(g+r)\}}$ represent the concatenation of the features of the first two and the last two streams, such that $RX_{\{g+r\}}=[RX_{gray\_ir},RX_{rgb\_ir}]$ and $RX_{\{LZM(g+r)\}}=[RX_{LZM(gray\_ir)},$ $RX_{LZM(rgb\_ir)}]$. The first row of Table 7 shows the results obtained using ResNet-50 for each stream of the proposed framework. We have computed the results given in the second row of the table by using $R101_{gray\_ir}$ and $R101_{rgb\_ir}$ for the third and fourth streams of the framework instead of ResNet-50 models that utilize the LZM pattern maps. In this way, there is a 0.8% improvement for Rank-1. However, the improvement becomes 1.7% when $R101_{gray\_ir}$ and $R101_{rgb\_ir}$ are replaced with $R101_{LZM(gray\_ir)}$ and $R101_{LZM(rgb\_ir)}$, as shown in the third row. Similarly, better results are achieved when $R152_{LZM(gray\_ir)}$ and $R152_{LZM(rgb\_ir)}$ are used for the third and fourth streams compared to using $R152_{gray\_ir}$ and $R152_{rgb\_ir}$, as shown in the fourth and fifth rows. The results of the experiments, where the ResNet-50 models are used for the first and second streams of the framework, are given in the first five rows of Table 7. The next five rows and the last five rows show the results obtained using ResNet-101 and ResNet-152 for the first two streams, respectively. Like the results given in the first five rows, these results also demonstrate that the LZM transformation plays an important role. Using the models trained with the LZM pattern maps, better performance improvements are achieved compared to the other models with the same depth and trained with grayscale & infrared and RGB & infrared images. This verifies that, by exposing the texture information from the images, the LZM transformation enables the ResNet architectures to learn different as well as complementary features.

4.3.1 Comparison With the State-Of-the-Art

Tables 8 and 9 show our results in comparison with the state-of-the-art. In [1], zero-padding is utilized to enable a one-stream network to learn domain-specific structures automatically. TONE+HCML [3] and TONE+XQDA [3] use a two-stage framework that includes feature learning and metric learning. BCTR [2] and BDTR [2] have a two-stream framework that employs separate networks for RGB and infrared images to extract domain-specific features. Kang et al. [5] train a one-stream network using single input images generated by placing visible and infrared images in different channels or by concatenating them. In [13], a generative adversarial network is used in order to extract common features for the images in different domains. D2RL [4] train a one-stream network with multi-spectral images generated by projecting RGB and infrared images to a unified space. Different from these works, in this study, we train multiple ResNet architectures by using the different representations of the input images and generate complementary features from each one of them for RGB and infrared images. As can be seen from Tables 8 and 9, we outperform the current state-of-the-art with a large margin. When we use ResNet-152 architectures in all the streams, our framework improves VI-ReId performance on SYSU-MM01 under all-search mode by $14.09\%/12.10\%$ and under indoor-search mode by $14.48\%/16.01\%$ in Rank-1$/$mAP. By concatenating all the feature vectors generated with ResNet-50, ResNet-101, and ResNet-152, we obtain at least 1.5% additional improvements for Rank-1 and mAP under both search modes. When we perform the re-ranking [10], the margin of the improvement further increases. We achieve $63.05\%/67.13\%$ and $69.06\%/76.95\%$ under all-search and indoor-search modes, respectively.

4.4 Re-Identification Performance on RegDB

In the experiments on the RegDB dataset, we employ only the ResNet-50 models due to the relatively low number of training images. As noted in Section 4.2, we train these models in two steps. In the first step, the training is carried out using a set created by combining SYSU-MM01 and RegDB training sets. Then, in the second step, the models are fine-tuned using only the images from the RegDB dataset. With such a training strategy, we aim to avoid overfitting.

According to the results given in Table 10, the models trained with LZM pattern maps exhibit lower performance than the others. As shown in Figures 1 and 2, LZM filters used perform high-pass filtering and reduce the low-frequency components of the images. This reduction results in LZM pattern maps to have less information compared to the original images. For this reason, in order to learn features with the same level of discriminating information (compared to using RGB or grayscale images), more training images will be needed in the training process performed with LZM pattern maps. In the first phase of the two-step training strategy, we expand the training set using the images from the SYSU-MM01 database. However, the images in the SYSU-MM01 and RegDB databases were recorded at different domains and so have different features from each other. Therefore, the models trained with the LZM pattern maps are not fed as many different images as they need in order to learn the specific features of the images in RegDB. As a result, models trained with the LZM pattern maps have lower performance than others. On the other hand, when the features obtained in all the streams of the proposed framework are combined, it is observed that Rank-1 and mAP are improved by 1.91% and 1.84%, respectively. This demonstrates that different and complementary features can be learned for also the RegDB dataset by using LZM pattern maps. Similar to [3], we have conducted additional experiments to evaluate the performance of our framework by following a different gallery/query setting where the RGB images are used as the gallery and the thermal images used as the query set. The results are given in Table 11. It is observed that the features of the models $R50_{LZM(rgb\_ir)}$ and $R50_{LZM(gray\_ir)}$ contribute to the performance by improving the Rank-1 and mAP by 2.32% and 2.22%, respectively.

4.4.1 Comparison With the State-Of-the-Art

In Table 12, our results are compared with the state-of-the-art. It is seen that we outperform the current state-of-the-art with a large margin by improving Rank-1 and mAP by $6.18\%$ and $11.06\%$, respectively. The improvements become $9.73\%$ and $16.36\%$ when we perform the re-ranking.

4.5 Re-Identification Performance on Market-1501

In this section, we have performed experiments on the Market-1501 dataset to expose the performance of the LZM transformation for the ReId problem in the visible domain. In these experiments, we use the ResNet-50 model and give the results in Table 13. The performance of ReId in the visible domain depends on the effective use of both low and high-frequency information of the images. However, as mentioned in the previous section, the LZM filters used perform high-pass filtering by reducing the low-frequency components. Therefore, lower performance is obtained with the features extracted from the LZM pattern maps. Results given in Table 13 indicate that the features of $R50_{LZM(gray\_ir)}$ and $R50_{LZM(rgb\_ir)}$ do not contribute to the performance obtained with the features of $R50_{gray\_ir}$ and $R50_{rgb\_ir}$.

4.6 Discussion on Execution Time and Memory Utilization

We have used the Chainer framework [61] for the implementation of the ResNet models used in this study and carried out the experiments on a PC with an Intel Core i7-4790 CPU (3.60GHz x 8), 32 GB RAM and an Nvidia GeForce GTX 1080Ti GPU. We have performed all the LZM calculations on the GPU. In this section, we show the execution time and the GPU memory utilization of the proposed framework by using ResNet-50 models.

$RX_{gray\_ir}$ and $RX_{rgb\_ir}$ models are trained using the input images with 3 channels. For the other models, $RX_{LZM(gray\_ir)}$ and $RX_{LZM(rgb\_ir)}$, the input images have 8 and 24 channels, respectively. Therefore, in this section, we initially compare the forward pass times of $R50_{rgb\_ir}$, $R50_{LZM(gray\_ir)}$, and $R50_{LZM(rgb\_ir)}$. The results are given graphically in Figure 6, where the graphs of $R50_{LZM(gray\_ir)}$ and $R50_{LZM(rgb\_ir)}$ show the total time spent on LZM transformation and ResNet-50. In Figure 6, it is seen that there is no significant difference in the forward pass times of the models until the batch size is four. When the batch size is five or more, the difference begins to occur. This is because the GPU utilization is less than 100% for all the three models when the batch size is four or less. After GPU utilization reaches 100%, delays occur in $R50_{LZM(gray\_ir)}$ and $R50_{LZM(rgb\_ir)}$ compared to the $R50_{rgb\_ir}$.

Figures 7(a) and 7(b) show forward pass time and GPU memory utilization when running 1, 2, 3, and 4 ResNet-50 models together on the same GPU. The last two of these models are $R50_{LZM(gray\_ir)}$ and $R50_{LZM(rgb\_ir)}$, respectively. According to Figure 7(a), when the batch size is one, the running of the four models increases the execution time by $1.7$ times compared to the running of a single model. This rate increases if the number of images in the batch increases. However, since there is no weight sharing between the models, each model can perform feature extraction independently. Therefore, the execution time can be easily pulled down using more GPUs. When the batch size is one, a single ResNet-50 needs 827 MB of memory, and this memory requirement increases to 2541 MB when the batch size is seven. As shown in Figure 7(b), the memory requirement is directly proportional to the number of models running together. Due to the lack of enough GPU memory, we were unable to run four models together for larger batch sizes.

5 Conclusion

In this study, we have introduced a four-stream framework for VI-ReId using ResNet architectures. In each stream of the framework, we train a ResNet by using a different representation of input images in order to obtain complementary features as much as possible from each stream. While grayscale and infrared input images are used to train the ResNet in the first stream, RGB and three-channel infrared images are used in the second stream. The first stream learns the features by using only the shape and texture information. The second stream uses the color information of the RGB images and learns to extract common features for the images with visually large differences. Unlike the first two streams, the input images in the other two streams are local pattern maps generated by employing the LZM transformation. Due to the lack of color, which provides the most important cues, in infrared images, the local shape and texture information is critical for VI-ReId. In the third and fourth streams, we expose this information from the images by generating the LZM pattern maps and train the ResNets using these maps as input images. With the exhaustive experiments performed employing three different ResNet architectures with different depths, we have demonstrated that each stream extracts different and complementary features and provides a significant contribution to the performance. Our framework outperforms, with a large margin, the current state-of-the-art on SYSU-MM01 and RegDB datasets. We further increase the improvement margin by utilizing re-ranking.