A survey of advances in vision-based vehicle re-identification

TL;DR

This survey comprehensively reviews vision-based vehicle re-identification methods, analyzing their performance on benchmark datasets, and discusses challenges and future research directions in the field.

Contribution

It systematically categorizes and evaluates vision-based V-reID methods, providing detailed analysis and insights into their strengths, weaknesses, and future trends.

Findings

Deep feature methods outperform hand-crafted features in accuracy.

Mean average precision varies significantly across methods.

Identified key challenges and potential future research directions.

Abstract

Vehicle re-identification (V-reID) has become significantly popular in the community due to its applications and research significance. In particular, the V-reID is an important problem that still faces numerous open challenges. This paper reviews different V-reID methods including sensor based methods, hybrid methods, and vision based methods which are further categorized into hand-crafted feature based methods and deep feature based methods. The vision based methods make the V-reID problem particularly interesting, and our review systematically addresses and evaluates these methods for the first time. We conduct experiments on four comprehensive benchmark datasets and compare the performances of recent hand-crafted feature based methods and deep feature based methods. We present the detail analysis of these methods in terms of mean average precision (mAP) and cumulative matching curve (CMC). These analyses provide objective insight into the strengths and weaknesses of these methods. We also provide the details of different V-reID datasets and critically discuss the challenges and future trends of V-reID methods.

Figures

Tables

Table 1. Table 1: Summary of sensor based methods for V-reID. We listed the strengths and weaknesses of magnetic sensors, inductive loop detectors, GPS, RFID, cellular phones, multi-sensor, and hybrid methods in the table.

Sensor based methods	Strengths	Weaknesses
Magnetic sensors	Can be used where loops are not viable	Requires pavement cut or tunneling
	(for example bridge decks).	under roadway.
	Less sensitive to stresses of traffic.	Cannot detect stopped vehicles
	Insensitive to inclement weather such as snow,	unless special sensor layouts or
	rain, and fog.	signal processing software are
		considered.
Inductive loop	Understood and well-known technology.	Poor installation reduces pavement
detectors	Provides different traffic parameters including	life.
	volume, presence, occupancy, speed, headway,	Multiple detectors usually required
	and gap.	to monitor a location.
	Resilient design to address large variety of	Installation and maintenance require
	applications.	lane closure.
GPS, RFID and	The GPS signal is available anywhere on	Susceptible to speed variations and
cellular phones	the globe.	environmental disturbances.
	The system gets calibrated by its own, and	In congested, two cars close to each
	therefore it is easy to be used.	other will likely be identified as a single
	Multiple lane operation available.	car.
Multi-sensor	Combine the strengths of many sensors.	Not capable of monitoring multiple
	Reliable even if traffic information is	lanes.
	incomplete.	Depend on constant vehicle speed.
	Models with small detection zones do not
	require multiple units for full lane detection.
Hybrid methods	Generally cost-effective in term of hardware	Performance affected by inclement
	requirements.	weather such as fog, rain, and snow.
	Easy to add and modify due to flexibiliy.	Performance also affected by vehicle
	Monitor multiple lanes.	shadows and vehicle projection.

Table 2. Table 2: Summary of vision-based methods for V-reID. Hand-crafted feature based methods and deep feature based methods published in different venues are listed. In the table, ’Authors collected’ means the dataset is collected by the same authors of the paper.

Category	Reference	Features/	Dataset	Performance	Venue	Year
		Model		Metric
	Woesler (2003)	3D Model	Wegedomstreet	Standard	IEEE ITSC	2003
		Color	Author collected	deviation
	Shan et al. (2005)	Edge based	Author collected	Hit rate	IEEE ICCV	2005
		model
	Guo et al. (2008)	3D Model	Author collected	Probablity of	IEEE CVPR	2008
		Appearance features		correct association
Hand-crafted	Hou et al. (2009)	Pose and	Author collected	Hit rate	IEEE CVPR	2009
features		illumination model
	Feris et al. (2012)	Motion	Authors	Hit rate	IEEE TMM	2012
		Shape	collected	False positive
	Zheng et al. (2015)	Individual paintings	Car dataset	Cumulative	IEEE MMSP	2015
		Matching		match curve
		3D Box	Authors	False positive	IEEE CVPRW	2016
	Zapletal and (2016)	Color histogram	collected
		String matching	Authors	Hit rate	IEEE AVSS	2017
	Watchar and (2017)		collected
	Liu et al. (2016c)	CNN	VeRi-776	Mean average	Springer ECCV
		SNN	Author collected	precision (mAP)
	Liu et al. (2016a)	Deep relative	VehicleID	Hit rate	IEEE CVPR	2016
		distance model	CompCars
	Liu et al. (2016b)	Texture	VeRi	Hit rate	IEEE ICME
		Color	Author collected	False positive
	Wang et al. (2017)	Siamese-CNN+	VeRi-776	mAP	IEEE CVPR
		Path-LSTM network	CompCars
	Shen et al. (2017)	Siamese-CNN+	VeRi-776	mAP	IEEE ICCV
		Path-LSTM network
Deep	Kanacı et al. (2017)	CNN	VehicleID	CMC	BMVC	2017
features				measure
	Zhang et al. (2017)	CNN	VeRi	mAP	IEEE ICME
	Tang et al. (2017)	CNN	VeRi	mAP	IEEE ICIP
	Li et al. (2017b)	Deep joint	VehicleID	mAP	IEEE ICIP
		discriminative model
	Liu et al. (2018)	Deep neural	VeRi	mAP	IEEE TMM
		network
	Zhou et al. (2018)	CNN-LSTM	BoxCars	mAP	IEEE TIP	2018
			Author collected
	Bai et al. (2018)	GSTE	PKU-Vehicle (Author)	mAP	IEEE TMM
			VehicleID, VeRi, CompCars

Table 3. Table 3: Summary of datasets. We listed 9 datasets for V-reID problem in terms of venue, publication year, number of images, number of vehicles, make, number of viewpoints, V-reID Anno, and availability. V-reID_Anno is the annotation of ids for V-reID task.

Datasets	Venue	Year	No. of	No. of	Images per	Make	No. of Viewpoints	V-reID_Anno	Availability
			Images	Vehicles	Vehicle
CompCars	CVPR	2015	2,14,345	1,687	127.05	161	$\sim$5	-	On request
VehicleID	IEEE CVPR	2016	2,21,763	26267	8.44	250	$\sim$2	$✓$	On request
VeRi	IEEE ICME	2016	40,000	619	64.62	-	$\sim$20	$✓$	On request
VeRi-776	Springer ECCV	2016	50,000	776	64.43	-	$\sim$20	$✓$	On request
BoxCars21k	IEEE CVPRW	2016	63,750	21,250	3.0	27	$\sim$4	$✓$	$✓$
BoxCars116k	IEEE ITS	2017	1,16,286	27496	4.22	45	$\sim$4	$✓$	$✓$
Toy Car ReID	IEEE TIP	2018	30,000	150	200	-	$\sim$8	$✓$	On request
VRID-1	IEEE ITS	2017	10,000	1000	10	10	$\sim$10	$✓$	On request
PKU-VD	IEEE ICCV	2017	1,904,909	2344	2	-	$\sim$2	-	On request

Table 4. Table 4: Computational complexity. Speed represents the complexity of each method in term of frames per seconds (fps). The complexity is presented for both deep feature (DF) based methods and hand-crafted feature (HCF) based methods.

DF based methods	Speed (fps)	HCF based methods	Speed (fps)
PROVID	08	3DCI	Real-time
DRDL	10	EMD	19
DCT	11	3DPM	13
OIM	09	3DPIM	18
VSTM	06	ABM	21
CLVR	15	MPM	Real-time
TWT	17	BBM	Real-time
FFM	14	LNP	Real-time
DJDL	12
NuFACT	09
MVF	13
GSTE	10

Equations

$$A=\frac{\sum _{l=1}^{n}S(l)\times c(l)}{M_c}$$

$$mAP=\frac{\sum _{u=1}^{V}A(u)}{V}$$

$$CMC=\frac{\sum _{u=1}^{V}c(u,l)}{V}$$

Full text

††journal: Computer Vision and Image Understanding

1 Introduction

In modern public transportation systems, video surveillance for traffic control and security plays a significant role. Therefore, obtaining accurate traffic information is increasingly in demand for many reasons including collection of statistical data de Dios OrtAozar and Willumsen (2011), controlling traffic signals and flows Lämmer and Helbing (2008) Ullah and Conci (2013) Khan et al. (2016) Ullah et al. (2017), law enforcement Vishnu et al. (2017), Maaloul et al. (2017) Saghafi et al. (2012) Hsu et al. (2013), smart transportation Zhang et al. (2011), accident detection Ullah et al. (2015), and urban computing Zheng et al. (2014). In traffic areas, many surveillance cameras are already installed. It would be advantageous to use these cameras for analysis of traffic scenes with no need of replacing them with some special hardware. The data from these cameras have been used tremendously to handle the problems of vehicle detection Ren et al. (2017) Zhou et al. (2016) Zha et al. (2007) Sun et al. (2006), tracking Ullah et al. (2016) Luo et al. (2019) Ullah and Alaya Cheikh (2018) and classification Yang et al. (2015) Chen et al. (2014) Zhang (2013). However, the problem of V-reID has escalated only over the past few years.

To re-identify a particular object, is to identify it as the same object as one observed on a previous occasion. When being presented with a vehicle of interest, V-reID tells whether this vehicle has been observed in another place by another camera. The problem of V-reID is to identify a target vehicle in different cameras with non-overlapping views as shown in Fig. 1. The emergence of V-reID can be attributed to 1) the increasing demand of public safety and 2) the widespread large camera networks in road networks, university campuses and streets. These causes make it expensive to rely solely on brute-force human labor to accurately and efficiently spot a vehicle-of-interest or to track a vehicle across multiple cameras.

V-reID research started with sensor based methods. Several important V-reID methods have been developed since then. This development included hybrid methods and computer vision based methods. We briefly depict V-reID history in Fig. 2. Recently, much attention has been focused on the development of V-reID methods. This is evidenced by an increasing number of publications in the different venues. In Fig. 3, the percentages of papers published in both categories: sensor based methods (first row) and vision based methods (second row) are presented.

In this survey, we walk through existing research on V-reID in the hope of shedding some light on what was available in the past, what is available now, and what needs to be done in order to develop better methods that are smartly aware of the traffic environment. To the best of our knowledge, comprehensive survey on V-reID is not available. This paper fills this gap by providing comprehensive summaries and analysis of existing V-reID methods and future trends. In this survey, we begin our discussion from sensor based to deep learning based methods. Therefore, our survey is spanned by comprehensive presentation of different works starting from 1990. It is worth noting that the main focus of our paper is vision based methods. However, we briefly categorize and discuss sensor based methods for the sake of completion. We focus on different V-reID methods currently available or likely to be visible in the future. We have given special emphasis to deep learning methods, which are currently popular topics or reflect future trends.

The rest of the paper is organized as follows. In Section 2, we present sensor based and vision based methods. The details of the datasets are presented in Section 3. Experiments and evaluation on three benchmark datasets considering 20 different methods are shown in Section 4. The challenges and future trends of V-reID are discussed in Section 5 and the conclusion is presented in Section 6.

2 Vehicle Re-identification Methods

In this section we consider sensor based and vision based methods. We divide sensor based methods into five categories: magnetic sensors, inductive loop detectors, GPS-RFID-cellular phones, multi-sensor, and hybrid methods. Vision based methods are divided into two categories: hand-crafted feature based and deep feature based methods. These categories are depicted in Fig. 4.

2.1 Sensor Based Methods

Sensor based methods use vehicle signatures Balke et al. (1995) Kuhne and Immes (1993) for V-reID. There are several hardware detectors to extract the vehicle signatures including inductive loop, infrared, ultrasonic, microwave, magnetic, and piezoelectric sensors Klein et al. (1997) Davies (1986). These methods estimate the travel time of individual vehicle by matching vehicle signature detected at one location (the upstream station) with the vehicle signature detected at another location (downstream station). The locations are separated by a few hundred meters. At each location, two sensors are installed in a speed-trap configuration as depicted in Fig. 5. The speed-trap configuration calculates the speed of each detected vehicle using its pair of signatures captured by the lead and lag sensors of the traveled lane. Next we discuss different sensor based methods individually.

2.1.1 Magnetic Sensors

Vehicles are composed of metallic masses that disrupt the Earth’s magnetic field. Therefore, magnetic sensors generate vehicles signatures that are different from one vehicle to the other. Magnetic sensors can provide detail temporal data related to the vehicle. This temporal data can be used in travel time estimation and V-reID process. Sanchez et al. (2011) proposed a wireless magnetic sensor to detect the vehicle signature. Charbonnier et al. (2012) proposed a single three-axis magnetic sensor to detect a tri-dimensional magnetic signature. They extract temporal features from the sensor and then train a Gaussian maximum likelihood classifier. Kwong et al. (2009) propose a wireless magnetic sensors to extract vehicle signatures for real-time estimation of travel time across multiple intersections. Yokota et al. (1996) used ultrasonic technology and Gimeno et al. (2013) used anisotrpic magneto-resistive (AMR) sensors for the V-reID. Kell et al. (1990) and Caruso and Withanawasam (1999) proposed other sensors including magnetoinductive probes also known as microloop and anisotropic magnetoresistive sensors, respectively. Kreeger and McConnell (1996) and Christiansen and Hauer (1996) proposed laser profile and weigh-in-motion axle profile (WIM) for the V-reID, respectively.

2.1.2 Inductive Loop Detectors

Inductive loop detectors (ILDs) are the most conventional ways for obtaining traffic data. They are widely deployed on major freeway networks. ILDs can provide various measurements like speed, length, volume, and occupancy information of a vehicle. Inductive loop signatures from vehicles are utilized by researchers Sun et al. (1999) Kuhne and Immes (1993) JENG and Chu (2013) Guilbert et al. (2013) Ndoye et al. (2008) Lunt Jeng (2007) Manual (2000) Dailey (1993) Kwon et al. (2000) to solve the problem of V-reID. Sun et al. (1999) formulates and solves the V-reID problem as a lexicographic optimization problem. JENG and Chu (2013) proposed a real-time inductive loop signature based V-reID method also called RTREID-2M, which is based on their previous method Jeng (2007) also called RTREID-2. Guilbert et al. (2013) track vehicle using inductive loop detector to obtain origin-destination matrix for the vehicle. Ndoye et al. (2008) and Lunt considered inductive loop data for extracting vehicle signature and estimated travel time for V-reID. Geroliminis and Skabardonis (2006) proposed an analytical model which estimates the travel time on signalized arterials based data. They utilized flow and occupancy information provided by the ILDs. Ndoye et al. (2009) and Oh et al. (2005) proposed a signal processing framework to improve travel time estimation for V-reID. Ali et al. (2013) proposed multiple inductive loop detector system for V-reID and lane change monitoring. V-reID using the ILDs are most sensitive to speed changes in between vehicle detection stations. These systems are based on the unrealistic assumption of constant speed.

2.1.3 GPS, RFID, and Cellular Phones

For V-reID, some methods explore beacon based vehicle tracking and monitoring systems for locating the position of the vehicles. These systems include global positioning systems (GPS), cell phones Smith et al. (2003), automatic vehicle identification (AVI) tags Zhou and Mahmassani (2006), radio frequency identification (RFID) tags Roberts (2006), and medium access control (MAC) addresses of blue tooth-enable devices CONTAIN (2008). The methods based on these systems are also called vehicle probe based methods Turner et al. (1998). Several methods related to cell phone based measurements for traffic monitoring are reported in the literature Alessandri et al. (2003) Astarita and Florian (2001) Bar-Gera (2007) Cheng et al. (2006) Liu et al. (2008) Smith et al. (2003) Ygnace et al. (2000) Yim and Cayford (2001) Saqib et al. (2011) Byeong-Seo et al. (2005) Wunnava et al. (2007). Prinsloo and Malekian (2016) proposed V-reID method based on RFID tags for electronic toll collection. Mazloumi et al. (2009) proposed GPS based method for vehicle travel time estimation. The aforementioned methods considered one sensor node (GPS, cell phones, AVI, and RFID) for measurement. A single sensor node is susceptible to speed variations and environmental disturbances Golob et al. (2004). Moreover, when the traffic flow is congested, two cars close to each other will likely be identified as a single, merged vehicle. Therefore, to determine precise vehicle status, some of researchers used multiple sensor nodes to deal with V-reID.

2.1.4 Multi-Sensor

The literature on multi-sensor fusion is extensive Waltz et al. (1990) Hall and McMullen (2004) Yang et al. (2014) Cho et al. (2014). There are many applications of multi-sensor data fusion including image fusion Hellwich and Wiedemann (1999) Sharma (1999)Hellwich and Wiedemann (2000), medical image analysis Leonard (1998), and intelligent transportation systems Klein (2000) Dailey et al. (1996). For V-reID, it was expected to get more accurate, reliable, and comprehensive information from multi-sensor fusion. Therefore, Ndoye et al. (2011) combine signal processing technique with multi-sensor data for V-reID. Tian et al. (2014) proposed a multi-sensor spatio-temporal correlation method for V-reID. The method relied on magnetic wireless sensor network. Several signature matching methods have been proposed by Zhang and Heidemann (2011) that uniquely identify vehicle signature by matching signatures coming from different sensors. Oh et al. (2007) used sensors in conjunction with inductive loop detectors. El Faouzi (2004) applied multi-sensor fusion to the problem of road travel time estimation. Cheu et al. (2001) proposed a neural network model (NNM) Rowley et al. (1998) for information fusion of traffic data. However, NNM requires a huge amount of data for training which is hardly available in engineering practices. Choi and Chung (2002) proposed a fusion algorithm based on fuzzy regression for travel time estimation. However, it is difficult to generalize fuzzy membership function fitting all the links of the road. To manage, analyze, and unify traffic data, linear estimation and weighted least square methods for information fusion are proposed by Zhang et al. (2005). To deal with incomplete and inaccurate traffic information coming from multi-sensor fusion, El Faouzi and Lefevre (2006) used evidence theory Yong et al. (2004). Kong et al. (2009) combine ILDs data with GPS data. They used federated Kalman filter Carlson and Berarducci (1994) and evidence theory to provide a robust and flexible fusion multi sensors data. However, El Faouzi and Lefevre (2006) and Kong et al. (2009) did not consider traffic signals which affect link travel time of vehicles. To address this problem, Bhaskar et al. (2008, 2009) estimated the cumulative number of vehicles plots on the upstream and downstream of a link. Hage et al. (2011) used unscented Kalman filter Wan and Van Der Merwe (2001) to estimate travel time in ubran areas by fusing data from ILDs with GPS sensor. Kerekes et al. (2017) proposed a multi-modal sensing method for vehicle classification and identification. They used electromagnetic emanations Vuagnoux and Pasini (2010), acoustic signatures Atalar (1979), and kernel regression Takeda et al. (2007) to classify and identify the vehicles. Multi-sensor methods are not capable of monitoring multiple lanes. They also depend on the assumption of constant vehicle speed. Therefore, hybrid methods were investigated that combine sensor techniques with computer vision techniques.

2.1.5 Hybrid Methods

Computer vision based methods are conveniently used in intelligent transportation systems. For example, Mallikarjuna et al. (2009) proposed video processing method for traffice data collection. Therefore, researchers proposed methods Wang et al. (2014)Ramachandran et al. (2002) Arr et al. (2004) Sun et al. (2004) for V-reID by combining image/video processing techniques with sensor data. Deng et al. (2017) proposed a V-reID method based on dynamic time warping Müller (2007) and magnetic signature. Wang et al. (2014) proposed a vehicle re-identification system with self-adaptive time windows to estimate the mean travel time for each time period on the freeway under traffic demand and supply uncertainty. Unlike inductive loop detectors, this method provide speed independent signature. Ramachandran et al. (2002) and Arr et al. (2004) proposed hybrid methods based on the signature captured from inductive loop detectors, vehicle velocity, traversal time, and color information based on images acquired from video cameras. Sun et al. (2004) fused data collected from inductive loop detector with color information captured from video camera.

Vehicle re-identification by matching vehicle signatures captured from inductive/ magnetic sensors are the most prominent, efficient and cost effective approaches. These approaches have several advantages over other methods. Firstly, these methods protect the privacy of the traveling public since vehicle signatures cannot be traced to individual vehicles. Secondly, probe penetration is 100% since no equipment inside the vehicle is needed. Finally, inductive/magnetic sensors are cost-effective. Despite these advantages, sensors based V-reID methods have several limitations. For example, these methods are not capable of monitoring multiple lanes. They are also having the limitation of constant speed constraint since they cannot provide speed independent signatures. The waveforms of vehicle signatures extracted from different sensors are cumbersome. Therefore, signatures matching algorithms are complex and depend on extensive calibrations. Moreover, sensor based methods cannot provide information about various features of the vehicles including color, length, and type.

Table 1 summarizes sensor based methods covered in this section in terms of their strengths and weaknesses.

2.2 Vision Based Methods

In computer vision, the goal of V-reID is to identify a target vehicle in multiple cameras with non-overlapping views. Due to the increase in traffic volumes on the roads and high demand in public safety, large camera network are mounted in different areas of parks, universities, streets, and other public areas. It is expensive and impractical to use traditional loop detectors or other sensors for the V-reID in such diverse environments. It is also a laborious job for security personnel to manually identify a vehicle of interest and track it across multiple cameras. Computer vision can automate the task of V-reID that can be broken down into two major modules. 1) Vehicle detection and 2) vehicle tracking through multiple cameras. Generally the first module is independent task and lots of energies have been poured to detect the vehicle in challenging and diverse environments. The challenging problem is V-reID, i.e., how to correctly match multiple images of the same vehicle under intense variations in appearance, illumination, pose, and viewpoint. The V-reID is considered as a multi camera tracking problem Javed et al. (2008) Wang (2013). We divided vision based methods into two categories: hand-crafted feature based methods and deep feature based methods.

2.2.1 Hand-Crafted Feature based methods

Hand-crafted features refer to properties derived using different methods that consider the information present in the image itself. For example, two simple features that can be extracted from images are edges and corners.

Many researchers considered appearance descriptors Woesler (2003) Shan et al. (2005) Khan et al. (2014) Ferencz et al. (2005) Shan et al. (2008) Feris et al. (2012) Khan (2014) Zheng et al. (2015) Zapletal and (2016) Liu et al. (2016b) for V-reID. In these methods, discriminatory information is extracted from the query vehicle image. Woesler (2003) extracted 3d vehicle models and color information from the top plane of vehicle for V-reID. Shan et al. (2005) proposed a feature vector which is composed of edge-map distances between a query vehicle image and images of other vehicles within the same camera view. A classifier was trained on the feature vectors extracted from the same and different vehicles. Ferencz et al. (2005) trained a classifier using image patches of the same and different vehicles. These image patches consisted of different features including position, edge contrast, and patch energy. Shan et al. (2008) proposed an unsupervised algorithm for matching road vehicles between two non-overlapping cameras. The matching problem is formulated as a same-different classification problem, which aims to compute the probability of vehicle images from two distinct cameras being from the same vehicle or different vehicle(s). Guo et al. (2008) and Hou et al. (2009) proposed 3D models for V-reID. They deal with large variations of pose and illumination in a better way. In the first step, pose and appearance of reference and target vehicles are estimated by 3D methods and in the second step vehicles are rendered in a normalized 3D space by making geometrically invariant comparisons of vehicles. Feris et al. (2012) proposed large-scale features using a feature pool containing hundreds of feature descriptors. Their method explicitly modeled occlusions and multiple vehicle types. Zheng et al. (2015) introduced multi-poses matching and re-ranking method for searching a car from a large-scale database. Watchar and (2017) proposed V-reID method based on license number plate Anagnostopoulos et al. (2008). They match vehicles through their license plate using cameras with automatic number plate recognition. Zapletal and (2016) also extracted 3D bounding boxes for V-reID. They used color histogram and histogram of oriented gradients followed by linear regressors. These methods Hou et al. (2009) Guo et al. (2008) Zapletal and (2016) are computationally expensive due to constraints of 3D models. Moreover, the performance of appearance based approaches is limited due to different colors and shapes of vehicles. Other cues such as license plates and special decorations might be hardly available due to camera view, low resolution, and poor illumination of the vehicles images. To cope with these limitations, deep feature based methods for V-reID are proposed.

2.2.2 Deep feature based methods

In recent years, the success of convolutional neural networks (CNNs) Krizhevsky et al. (2012) Simonyan and Zisserman (2014) Szegedy et al. (2015) Ullah et al. (2018) in different computer vision problems has inspired the research community to develop CNN based methods for vehicle recognition Hu et al. (2015) Ramnath et al. (2014), vehicle categorization and verification Yang et al. (2015), object categorization Krause et al. (2013) and recognition Xiang et al. (2015).

Liu et al. (2016b) considered large-scale bounding boxes for V-reID. They combine color, texture, and high level semantic information extracted by deep neural network. For V-reID, Liu et al. (2016c) proposed two networks: a convolutional neural network (CNN) for learning appearance attributes and a siamese neural network (SNN) for the verification of license number plates of vehicles. For learning appearance attributes, they adopted a fusion model of low-level and high-level features to find similar vehicles. The SNN was used to verify whether two license plates images belong to the same vehicle. This network is trained with large number of license number plate images for the verification. The SNN is based on deep distance metric learning Song et al. (2016) and it simultaneously minimizes the distances of similar object pairs and maximizes the distances among different object pairs. The SNN was initially proposed by Bromley et al. (1994) for the verification of hand written signatures. It was also adopted by Chopra et al. (2005) for face verification and Zhang et al. (2016) for gait recognition for human identification. Liu et al. (2016a) proposed deep relative distance learning (DRDL) method for V-reID. They aimed to train a deep convolutional neural network considering a triplet loss function to speed up the training convergence. Their model takes raw images of vehicles and project them into a Euclidean space where L2 distance can be used directly to measure the similarity between two or more arbitrary vehicle images. The key idea of the DRDL was to minimize the distance between the arbitrary views of the same vehicle and maximize those of other vehicles. Zhang et al. (2017) proposed triplet-wise training of convolutional neural network (CNN). This training adopts triplets of query, positive example, and negative example to capture the relative similarity between vehicle images to learn representative features. They improved the triplet-wise training at two ways: first, a stronger constraint namely classification-oriented loss is augmented with the original triplet loss; second, a new triplet sampling method based on pairwise images is modeled. Li et al. (2017b) proposed a deep joint discriminative learning (DJDL) model, which extracts discriminative representations for vehicle images. To exploit properties and relationship among samples in different views, they modeled a unified framework to combine several different tasks efficiently, including identification, attribute recognition, verification and triplet tasks. The DJDL model was optimized jointly via a specific batch composition design. Tang et al. (2017) investigated that deep features and hand-crafted features are in different feature space, and if they are fused directly together, their complementary correlation is not able to be fully explored. Therefore, they proposed a multi-modal metric learning architecture to fuse deep features and hand-crafted features in an end-to-end optimization network, which achieves a more robust and discriminative feature representation for V-reID. Cui et al. (2017) proposed a deep neural network to fuse the classification outputs of color, model, and pasted marks on the windshield. They map them into a Euclidean space where distance can be directly used to measure the similarity of arbitrary two vehicles. Kanacı et al. (2017) proposed a CNN based method that exploits the potentials of vehicle model information for V-reID. The method avoids expensive and time consuming cross-camera identity pairwise labeling and relies on cheaper vehicle model. Shen et al. (2017) proposed a two stage framework that takes into account the complex spatio-temporal information. The method takes a pair of vehicle images with their spatio-temporal information. Each image is associated with three types of information, i.e., visual appearance, time stamp, and geo-location of the camera. By employing MRF model, a candidate visual-spatio-temporal path is generated, where each visual-spatio-temporal state corresponds to actual image with its visual-spatio-temporal information. A siamese-CNN+Path-LSTM is employed that takes candidate path and image pair and compute their similarity score. Wang et al. (2017) proposed a method that incorporates orientation invariant feature with spatial-temporal information for V-reID. The method consists of two main components: the orientation invariant feature and spatial-temporal regularization component. The method starts by feeding vehicle images into a region proposal module, which computes response maps of 20 key points. The key points are then clustered into four orientation base region proposal masks. A global feature vector and four region vectors are then generated through a learning module. These features are fused together through an aggregation module that gives final orientation invariant feature vector. The second component models the spatio-temporal relations between the query and gallery images. Liu et al. (2018) introduced a Null space based fusion of color and attribute feature model. They adopted a Null Foley-Sammon transform (NFST) Guo et al. (2006) based metric learning method for the fusion of features. Their model learns discriminative appearance features from multiple and arbitrary view points and reduces dimensionality of the feature space. This method utilized time and geo-location information for each query image and spatio-temporal information are exploited for every pair of images. The method works well for the pair of images that are spatially and temporally close to each other. Zhou et al. (2018) addressed multi-view V-reID problem by generating multi-view features for each query image which can be considered as a descriptive representation containing all information from the multiple views. The method extracts features from one image that belong to one view. Transformation models are then learned to infer features of other viewpoints. Finally, all the features from multiple viewpoints are fused together and distant metric learning is used to train a network. For the purpose of inferring features from hidden viewpoint, two end-to-end networks, spatially concatenated convolutional network and CNN-LSTM bi-directional loop (CLBL) are proposed. Bai et al. (2018) proposed a deep metric learning method, group sensitive triplet embedding (GSTE), to recognize and retrieve vehicles, in which intra-class variance is magnificently modeled by using an intermediate representation “group” between samples and each individual vehicle in the triplet network learning. To acquire the intra-class variance attributes of each individual vehicle, they utilized online grouping method to partition samples within each vehicle ID into a few groups, and build up the triplet samples at multiple granularities across different vehicle IDs as well as different groups within the same vehicle ID to learn fine-grained features.

Table 2 summarizes vision based methods covered in this section in terms of feature/model, dataset, performance metric, venue, and publication year.

3 Datasets

In order to explore V-reID problem, several methods and datasets have been introduced during past couple of years. The V-reID problem not only faces challenges of enormous intra-class and least inter-class differences of vehicles but also suffers from complicated environmental factors including changes in illumination, viewpoints, scales, and camera resolutions. Therefore, in order to develop robust V-reID methods, it is important to acquire data that captures these factors effectively. A dataset should consist of sufficient amount of data so that a V-reID model can learn intra-class variability. It should also consist of huge amount of annotated data collected from a large camera network. In order to address these challenges, attempts have been made in this direction and a few datasets are collected. We discuss them individually in detail.

3.1 CompCars

Yang et al. Yang et al. (2015) collected CompCars dataset. It is large scale and comprehensive vehicle dataset with 214,345 images of 1,716 car models. The dataset is labelled with five viewpoints including front, rear, side, front-side, and rear side. This dataset also consists of car parts as well as other attributes such as seat capacity, door number, and type of car. The images are collected from web and urban surveillance camera. Most of the images are collected from the web which cover different viewpoints and 50,000 images are captured from the surveillance camera that cover only front view. Each image is annotated with bounding box, model, and color of the car. This dataset offers four unique and important features. 1) Car hierarchy where car models are organized into a hierarchical tree of three layers, namely, car make, car model, and year of manufacturing. 2) Car attributes where each car is labeled with five attributes, namely, maximum speed, displacement, number of doors, number of seats, and type of car. 3) Viewpoints where each car is labeled with five view points. 4) Car parts where each car is labeled with eight car parts, namely, headlight, taillight, fog light, air intake, console, steering wheel, dashboard, and gear lever. This dataset was originally collected for car classification, attributes prediction, and car verification. In Fig. 6, sample images from CompCars dataset are depicted.

3.2 VehicleID

Liu et al. (2016a) collected VehcileID dataset which consists of 2,21,763 images of 26,267 cars captured from multiple non-overlapping surveillance cameras. The dataset contains car images captured from only two viewpoints (front and back) and the information about the other viewpoints is not provided. All car images are annotated with ID numbers indicating correct identities according to the car’s license plate. Additionally, 90,000 images of 10,319 vehicles are also annotated with the vehicle model information. This dataset can be used for vehicle model verification, vehicle retrieval, and V-reID problems. In Fig. 7, sample images from VehicleID dataset are presented.

3.3 BoxCars

Sochor et al. (2016) collected BoxCars dataset from 137 surveillance cameras. This dataset consists of two variants: BoxCars21K and BoxCars116K. The first variant BoxCars21K contains 63,750 images of 21,250 vehicles of 27 different models. The second variant BoxCars116K contains 1,16,826 images of 27,496 vehicles of 45 different models. The dataset contains vehicle images captured from arbitrary viewpoints, i.e., front, back, side, and roof. All vehicle images are annotated with 3D bounding box, make, model, and type. Vehicle type is annotated by tracking each vehicle across multiple cameras. Each correctly detected vehicle has three images in BoxCars21K which has been extended to 4 images per track. BoxCars dataset is designed for fine-grained vehicle model, make classification, and recognition. The images in the dataset are arranged according to the vehicle identities. Therefore, it can be also be used for V-reID problem. In Fig. 8, sample images from BoxCars dataset are shown.

3.4 VeRi-776

Liu et al. (2016c) collected VeRi-776 dataset which is the extension of VeRi dataset collected by Liu et al. (2016b). All the vehicle images in this dataset are captured in natural and unconstrained traffic environment. The dataset is collected from 20 surveillance cameras with arbitrary orientations and tilt angles. Most of the scenes include two lane roads, four lane roads, and cross roads. The dataset consists of 50,000 images of 776 cars, where each image of a vehicle is captured from 2$\sim$18 viewpoints with different illuminations and resolutions. The vehicles are labeled with bounding boxes over whole vehicle body, type, color, and cross camera vehicle correlation. Moreover, vehicle license plate and spatial temporal relations are also annotated for tracks of all vehicles. Due to high recurrence rate and with large number of vehicles images captured with diverse attributes, the dataset is widely used in the V-reID problem. In Fig. 9, different vehicles, from VeRi-776 dataset, in terms of color, type, model, and different viewpoints are shown.

3.5 Toy Car ReID

Zhou et al. (2018) collected Toy Car ReID dataset. This is the first synthetic vehicle dataset collected in an indoor environment using multiple cameras. The dataset contains 200 metal toy cars of common vehicle types including sedan, SUV, hatchback, van, and pickup. The dataset is constructed with only those metal toy cars which have close resemblance with their real counter parts. In addition, changes in lighting are also incorporated to simulate the illumination changes by sun. In order to obtain densely sampled viewpoints, each vehicle is rotated by 360 degrees. The cameras are set at three different angles: 30, 60, and 90 degrees to capture images with different altitudes. In each angle, vehicle images are sampled from 50 viewpoints and then cropped to generate dataset comprising of 30,000 images. Sample images from Toy Car ReID dataset are shown in Fig. 10.

3.6 VRID-1

Li et al. (2017a) collected vehicle re-identification dataset-1 (VRID-1). This dataset consists of 10,000 images captured in daytime. There are 1,000 individual vehicles. For each vehicle model, there are 100 individual vehicles, and for each of these, there are ten images captured at different locations. The images in VRID-1 are captured using 326 surveillance cameras. Therefore, there are various vehicles poses and levels of illumination. This dataset provides images of good enough quality for the evaluation of V-reID methods.

3.7 PKU-VD

The PKU-VD dataset collected by Yan et al. (2017) consists of two subdatasets VD1 and VD2 based on real-world unconstrained scenes from two cities, respectively. The images in VD1 (total 1,097,649 images) are captured from high resolution traffic cameras, and images in VD2 (total 807,260 images) are obtained from surveillance videos. For each image in both two datasets, the authors provide different annotations including identity number, precise vehicle model, and vehicle color. In the dataset, identity number is unique and all images belong to the same vehicle have the same ID. Additionally, the authors annotated 11 common colors in the dataset. Sample images from PKU-VD dataset are depicted in Fig. 11.

Table 3 summarizes the datasets covered in this section in terms of venue, publication year, number of images, number of vehicles, make, number of viewpoints, V-reID Anno, and availability.

4 Experiments and Evaluation

For experimental analysis, we consider 8 hand-crafted feature based methods and 12 deep feature based methods. These are recently published methods in both categories. The 8 hand-crafted feature based methods are: the 3d and color information (3DCI) Woesler (2003), the edge-map distances (EMD) Shan et al. (2005), the 3d and piecewise model (3DPM) Guo et al. (2008), the 3d pose and illumination model (3DPIM) Hou et al. (2009), the attribute based model (ABM) Feris et al. (2012), the multi-pose model (MPM) Zheng et al. (2015), the bounding box model (BBM) Zapletal and (2016), and the license number plate (LNP) Watchar and (2017). The 12 deep feature methods are: the progressive vehicle re-identification (PROVID) Liu et al. (2016c), the deep relative distance learning (DRDL) Liu et al. (2016a), the deep color and texture (DCT) Liu et al. (2016b), the orientation invariant model (OIM) Wang et al. (2017), the visual spatio-temporal model (VSTM) Shen et al. (2017), the cross-level vehicle recognition (CLVR) Kanacı et al. (2017), the triplet-wise training (TWT) Zhang et al. (2017), the feature fusing model (FFM) Tang et al. (2017), the deep joint discriminative learning (DJDL) Li et al. (2017b), the Null space based Fusion of Color and Attribute feature (NuFACT) Liu et al. (2018), the multi-view feature (MVF) Zhou et al. (2018), and the group sensitive triplet embedding (GSTE) Bai et al. (2018).

To assess the performance of the aforementioned V-reID methods, we select four comprehensive datasets including VeRi-776, VehicleID, CompCars, and PKU-VD. This selection is based on the available attributes of these datasets. These datasets consist of large scale vehicle models with multiple views. Therefore, they present reasonable information for proper comparison of different V-reID methods. Furthermore, we consider two widely used V-reID performance metrics: the mean average precision (mAP) and the cumulative matching curve (CMC) for quantitative evaluation. The mAP metric evaluates the overall performance for V-reID. The average precision is computed for each query image as formulated in Eq. 1,

[TABLE]

where $l$ is the rank in the order of retrieved vehicles, $n$ is the number of fetched vehicles, $M_{c}$ is the number of relevant vehicles. $S(l)$ is the precision at cut-off $l$ in the recall list and $c(l)$ shows whether the $l-th$ recall image is correct. Therefore, the mAP is formulated in Eq. 2,

[TABLE]

where $V$ is the number of total query images. The CMC indicates the probability that a query identity occurs in different-sized candidate list. The CMC at rank k can be formulated as in Eq. 3,

[TABLE]

where $c(u,l)$ equals 1 when the ground truth of $u$ image occurs before rank k. The CMC evaluation is valid only if there is only one ground truth match for a given query image.

For proper comparison, we divide the training and testing samples of each dataset according to the standard protocol. VeRi-776 dataset consists of 50,000 images of 776 vehicles, in which each vehicle is captured by 2-18 cameras in different viewpoints, illuminations, resolutions and occlusions. The training set has 576 vehicles with 37,781 images and the testing set has 200 vehicles with 11,579 images. VehicleID dataset consists of 221,763 images of 26,267 vehicles. There are 110,178 images of 13,134 vehicles for training and 111,585 images of 13,133 vehicles for testing. CompCar dataset contains 136,727 vehicle images of 1687 different vehicle models. We choose the Part-I subset for training that contains 16,016 images of 431 vehicle models and the remaining 14,939 images for test. For PKU-VD dataset, we split it into training and testing sets according to Yan et al. (2017) scheme. To build the training set, we randomly choose nearly half of vehicles from each vehicle model to construct the training set. The rest of the vehicles are used to build the testing set. The numbers of intra-class groups in CompCar, VeRI-776 and VehicleID are empirically set to be 5, 5 and 2, respectively. Learning rate is divided by10 every 15 epoches and the models are trained for 50 epoches.

In Fig. 12, we present the mAP results for the four datasets for all the methods. The hand-crafted feature based methods are annotated in red and the deep feature based methods are annotated in blue. The first, second, third, and last row list the mAP results on VeRi-776, VehicleID, CompCars, and PKU-VD datasets, respectively. The experimental results show that the GSTE Bai et al. (2018), NuFACT Liu et al. (2018), OIM Wang et al. (2017), DRDL Liu et al. (2016a), and VSTM Shen et al. (2017) outperform the rest of the methods. They are the latest available state-of-the-art methods for V-reID. The gain of the performance of the GSTE Bai et al. (2018) originates from the GSTE loss function, such that GSTE should be able to generalize to other network structures. In fact, the deeper networks of GSTE learn better feature representation. From the comparison, the GSTE loss based network outperforms the hand-crafted feature based methods significantly. The improvements across networks suggest that GSTE is generic work with the deep network structure to present consistently better performance in V-reID. The NuFACT Liu et al. (2018) utilizes the multimodality features of the data. The method also considers coarse-to-fine search in the feature domain and near-to-distant search in the physical space. The orientation invariant feature embedding module of the OIM Wang et al. (2017) handle multiple views of a vehicle efficiently. The DRDL Liu et al. (2016a) exploits a two-branch deep convolutional network to map vehicle images into a Euclidean space. Their coupled clusters loss function and the mixed difference network structure perform key role in achieving a high predict accuracy. The VSTM Shen et al. (2017) method based on the visual-spatio-temporal path proposal does provide vital priors for robustly estimating the vehicle similarities.

In Fig. 13, we present the CMC curves for the four datasets for all the methods. The first, second, third, and last row depict the CMC curves of VeRi-776, VehicleID, CompCars, and PKU-VD datasets, respectively. All the four rows show that the GSTE Bai et al. (2018), NuFACT Liu et al. (2018), OIM Wang et al. (2017), DRDL Liu et al. (2016a), and VSTM Shen et al. (2017) outperform the rest of the methods. The best performance of GSTE Bai et al. (2018) in the first and last rows shows that the introduction of the intra-class variance structure and its relevant loss function to triplet embedding bring significant improvement over the other methods. Moreover, we can observe that considering large data and deep network structure can ensure more effective network training, and generate more discriminative feature representation for fine-grained recognition. The OIM Wang et al. (2017) outperforms all the other methods in the second row showing the results on VehicleID dataset. The OIM extracts local region features of different orientations based on 20 key point locations. They retrieve refined result using the log-normal distribution to model the spatial-temporal constraints. In the third row, the NuFACT Liu et al. (2018) surpasses all the other methods considering the CompCars dataset. Thus NuFACT achieves much better improvement on CompCars than VeRi-776 and VehicleID. In fact, the fusion of color feature with semantic attributes can work better on CompCars. Moreover, during training of the null space, more information can be learned on CompCars. Thus, the NuFACT achieves greater improvement.

Both the mAP results in Fig. 12 and the CMC curves in Fig. 13 show large differences in the overall performances of hand-crafted feature based methods and deep feature based methods. The deep learning empowers the creation of complex networks, where deep layers act as a set of feature extractors that are often quite generic and, to some extent, independent of any specific task at hand. This means that deep learning extracts a set of features learned directly from observations of the input images. The clue is to discover multiple levels of representation so that higher level features can represent the semantics of the data, which in turn can provide greater robustness to intra-class variability.

To calculate the computational complexity, a 16GB RAM computer with a 4.20 GHz CPU and a powerful NVIDIA GPU are used to perform the experiments. Further reduction in the computational complexities is possible since these implementations are not optimized. In Table 4, we provide the computational complexities for both deep feature (DF) based methods and hand-crafted feature (HCF) based methods. These complexities are presented in term of average frames per second (fps) calculated over all the datasets. Deep feature based methods are computationally more complex than hand-crafted feature based methods. Therefore, they execute less frames per seconds.

5 Challenges and Future Trends of V-reID

V-reID is a broad and difficult problem with numerous open issues Gong et al. (2014). In this section we discussed the general problems of V-reID and its broader challenges.

Inter- and Intra-class variations: A basic challenge of any V-reID method is to cope with the inter-class and intra-class variations. Considering inter-class variation, different vehicles can look alike across camera views. Two vehicles from the same viewpoint always look more alike than two different viewpoints of one vehicle by machine vision. Considering intra-class variation, the same vehicle from different sides looks different when observed under different camera views. Such variations between camera view pairs are in general complex and multi-modal. Visual features of vehicles with varying viewpoints from different cameras have significant differences. There is no overlap of the visual features for a vehicle across the front, side, rear, and top viewpoints. The chassis of a vehicle is not upright like a person, thus the texture or color will change severely in different views. Therefore, the appearance variation across different viewpoints of a vehicle is much larger. Additionally, different views of a vehicle from multiple cameras leads to deformable shapes of silhouettes and different geometries.

Data requirements: In general a V-reID method may be required to match single vehicle image to a gallery of images. This means there is likely to be insufficient data to learn a good model of each vehicleś intra-class variability. The datasets we discussed hardly reflect the challenges of real world surveillance. For example, vehicle has to be tracked through a large number of cameras with overlapping and non-overlapping views in real world surveillance. The currently available datasets are collected from non-overlapping views with limited number of cameras, which capture less variations in resolution and viewpoints. Due to unavailability of unconstrained data, the impact of integrating and exploiting temporal information can not considered. However, this type of information is important for learning inter-camera relations which can increase the efficiency of V-reID methods by suppressing the false positives. Moreover, multiple vehicles re-identifications are not considered in the current datasets.

Generalization capability: This is the other side of the training data scalability. If a model is trained for a specific pair of camera, it does not generalize well to another pair of cameras with different viewing conditions. Good generalization ability of a model is generally desirable that can be trained once and then applied to a variety of different camera configurations from different locations. This would avoid the issue of training data scalability.

Long-term V-reID: The longer the time and space separation between views, the greater the chance that vehicles may appear with some changes. In fact, the nature of separation between the cameras determines the level of difficulty in V-reID system. For example, the V-reID modeling could be easy if the two images are taken from similar views only a few minutes apart. However, if the images/video are taken from different views hours apart, the modeling could not be easy due to variations in illumination and view angles. This highlights the sensitive nature of the V-reID modeling. The temporal segregation between the images is a key factor in the complexity of the V-reID system. Therefore, a V-reID method should have some robustness. The current datasets cannot address long-term V-reID problem due to the unavailability of long duration videos, recorded over several days using the same or different set of cameras.

Other challenges: To develop a V-reID model, features are extracted from an image input or a video input. In the former case, a V-reID system detects and localize a vehicle in an image. In the latter case, the system establishes correspondence between the detected vehicles across multiple frames to ensure that the features belong to the same vehicle of interest. This procedure is called tracking that caters a consistent label to each vehicle in multiple frames. The V-reID system uses multiple instances of a vehicle for feature extraction and subsequent descriptor generation for V-reID. However, vehicle detection and multiple vehicle tracking are different problems with their own challenges. Moreover, it is difficult to model discriminative visual descriptors since vehicles can be partially or completely occluded due to crowded traffic or clutter. It is also difficult to control factors such as resolution, frame rate, imaging conditions, and imaging angles. Therefore, extracting a unique and discriminative descriptor is dependent upon availability of good quality observations. Additionally, the quality of the V-reID system is affected due to incorrect detection and trajectory estimation.

Despite these challenges, the performance of deep feature based methods is noticable. One of the main benefits of deep learning over other methods is its ability to generate new features from limited series of features located in the training dataset. These methods use more complex sets of features in comparison with hand-crafted feature based methods. In fact, deep learning generates actionable results when solving the V-reID problem. The performance of deep feature based methods can be further improved if large amount of training data is provided.

6 Conclusions

In this paper we presented the problem of vehicle re-identification and an overview of current research in the computer vision community. We discussed sensor-based and vision based methods in detail. We categorized vision based methods into hand-crafted feature based methods and deep feature based methods. The details of different datasets are presented and summarized. We also presented results on three benchmark datasets considering 20 different V-reID methods including both hand-crafted feature based and deep feature based methods. We highlighted open issues and challenges of the V-reID problem with the discussion on potential directions for further research.

In our future work, we would collect large scale real surveillance multi-view vehicle datasets to improve the training of the state-of-the-art models for performance enhancement.

Acknowledgments

This work is supported by the University of Hail, Saudi Arabia. Portions of the research in this paper use the PKU-VD dataset collected under the sponsor of the National Basic Research Program of China and the National Natural Science Foundation of China.