DensePeds: Pedestrian Tracking in Dense Crowds Using Front-RVO and Sparse Features

TL;DR

DensePeds is a novel pedestrian tracking algorithm designed for dense crowds, combining a new motion model and sparse features to improve accuracy and speed in challenging scenarios.

Contribution

Introduces Front-RVO, a new collision avoidance-based motion model, and integrates it with Mask R-CNN for enhanced dense crowd pedestrian tracking.

Findings

4.5 times faster than previous methods on MOT benchmark

Achieves over 2.6% improvement in dense crowd tracking accuracy

Performs well on both standard and new dense crowd datasets

Abstract

We present a pedestrian tracking algorithm, DensePeds, that tracks individuals in highly dense crowds (greater than 2 pedestrians per square meter). Our approach is designed for videos captured from front-facing or elevated cameras. We present a new motion model called Front-RVO (FRVO) for predicting pedestrian movements in dense situations using collision avoidance constraints and combine it with state-of-the-art Mask R-CNN to compute sparse feature vectors that reduce the loss of pedestrian tracks (false negatives). We evaluate DensePeds on the standard MOT benchmarks as well as a new dense crowd dataset. In practice, our approach is 4.5 times faster than prior tracking algorithms on the MOT benchmark and we are state-of-the-art in dense crowd videos by over 2.6% on the absolute scale on average.

Tables

Table 1. TABLE I: Notations we use in this paper.

$p_i$	$i^{\text{th}}$ pedestrian
$h_j$	$j^{\text{th}}$ detected pedestrian
$f_{p_i}$ & $f_{h_j}$	feature vectors corresponding to $p_i$ and $h_j$ respectively
$T$	total number of frames in the video
$\mathscr{H}$ & $𝒫$	sets of all pedestrian detections and total pedestrians in a frame respectively
$\rho$	predefined radius around every $p_i$
${\mathscr{H}}_i$	set of all detected persons within a circle of radius $r$ centered around $p_i$
$N_i$	cardinality of the set ${\mathscr{H}}_i$
$d(a,b)$	cosine distance metric between the vectors $a$ and $b$, defined as: $d(a,b)=1-\frac{a^{\top }b}{\parallel a\parallel \parallel b\parallel }$

Table 2. TABLE II: Evaluation on our dense crowds dataset with MOTDT  [ 39 ] and MDP  [ 40 ] . MOTDT is currently the best online tracker on the MOT benchmark with open-sourced code. Bold is best. Arrows ( ↑ , ↓ ↑ ↓ \uparrow,\downarrow ) indicate the direction of better performance. Observation: DensePeds improves the accuracy (MOTA) over the state-of-the-art by 2.6%. We reduce the number of false negatives (FN) by 11% compared to the next best method, which is a direct consequence of the theoretical formulation in Section  III-B .

Sequence Name	Tracker	MT(%)$\uparrow$	ML(%)$\downarrow$	IDS$\downarrow$	FN$\downarrow$	MOTP(%)$\uparrow$	MOTA(%)$\uparrow$
IITF-1	MOTDT	1.8	81.1	84 (0.2%)	10,050 (29.3%)	61.2	70.5
	MDP	1.8	81.1	40 (0.1%)	10,287 (30.0%)	61.2	69.9
	DensePeds	1.8	43.4	194 (0.5%)	8,357 (24.3%)	60.7	75.1
IITF-2	MOTDT	5.7	40.0	51 (0.3%)	3,902 (23.6%)	70.9	76.1
	MDP	14.2	25.7	36 (0.2%)	3,033 (18.3%)	72.6	81.4
	DensePeds	17.1	11.5	82 (0.5%)	2,308 (14.0%)	70.3	85.5
IITF-3	MOTDT	2.0	12.5	187 (0.5%)	7,563 (19.4%)	69.5	80.1
	MDP	8.3	16.6	85 (0.2%)	7,735 (19.8%)	71.0	79.9
	DensePeds	22.9	4.2	204 (0.5%)	5,968 (15.3%)	69.0	84.2
IITF-4	MOTDT	11.4	17.2	83 (0.5%)	3,025 (19.2%)	68.2	80.2
	MDP	14.2	28.5	41 (0.3%)	3,129 (19.9%)	70.3	79.8
	DensePeds	14.2	5.8	114 (0.7%)	2,359 (15.0%)	67.6	84.3
NDLS-1	MOTDT	0	78.3	113 (0.3%)	9,965 (28.9%)	61.8	70.8
	MDP	0	89.1	72 (0.2%)	10,293 (29.8%)	62.0	70.0
	DensePeds	0	76.1	131 (0.4%)	9,728 (28.2%)	61.5	71.4
NDLS-2	MOTDT	0	85.2	142 (0.3%)	14,084 (28.9%)	63.0	70.8
	MDP	0	85.1	82 (0.2%)	13,936 (28.6%)	63.9	71.2
	DensePeds	0	74.5	183 (0.4%)	13,240 (27.2%)	62.7	72.5
NPLACE-1	MOTDT	1.9	64.7	181 (0.5%)	9,676 (27.7%)	63.3	71.8
	MDP	1.9	64.7	153 (0.4%)	9,629 (27.5%)	63.1	72.0
	DensePeds	1.9	51.0	210 (0.6%)	9,114 (26.1%)	62.8	73.3
NPLACE-2	MOTDT	3.8	67.3	83 (0.3%)	7,443 (27.4%)	63.2	72.3
	MDP	5.7	69.2	54 (0.2%)	7,484 (27.6%)	62.8	72.2
	DensePeds	3.8	61.6	90 (0.3%)	7,161 (26.4%)	62.8	73.3
Summary	MOTDT	2.9	58.0	924 (0.4%)	65,708 (26.2%)	66.1	73.4
	MDP	5.1	59.9	563 (0.2%)	65,526 (26.1%)	67.4	73.7
	DensePeds	7.0	43.4	1,208 (0.5%)	58,235 (23.2%)	65.7	76.3

Table 3. TABLE III: Evaluation on the MOT benchmark with non-anonymous, online methods. In the interest of a fair comparison, we have chosen methods that have an average rank either greater than or equal to our average rank. Bold is best. Arrows ( ↑ , ↓ ↑ ↓ \uparrow,\downarrow ) indicate the direction of better performance. Observation: DensePeds is up to 4.5 × \times faster than previous methods and improves accuracy by up to 17%. Its overall performance has a rank of 23.5 (top 13%) of all published methods on the MOT15 and 28.8 (top 20%) on MOT16.

	Tracker	Hz$\uparrow$	MT(%)$\uparrow$	ML(%)$\downarrow$	IDS$\downarrow$	FN$\downarrow$	MOTP(%)$\uparrow$	MOTA(%)$\uparrow$
MOT15	AMIR15 [41]	1.9	15.8	26.8	1026	29,397	71.7	37.6
	HybridDAT [42]	4.6	11.4	42.2	358	31,140	72.6	35.0
	AM [17]	0.5	11.4	43.4	348	34,848	70.5	34.3
	AP_HWDPL_p [43]	6.7	8.7	37.4	586	33,203	72.6	38.5
	DensePeds	28.9	18.6	32.7	429	27,499	75.6	20.0
MOT16	EAMTT_pub [44]	11.8	7.9	49.1	965	102,452	75.1	38.8
	RAR16pub [45]	0.9	13.2	41.9	648	91,173	74.8	45.9
	STAM16 [17]	0.2	14.6	43.6	473	91,117	74.9	46.0
	MOTDT [39]	20.6	15.2	38.3	792	85,431	74.8	47.6
	AMIR [41]	1.0	14.0	41.6	774	92,856	75.8	47.2
	DensePeds	18.8	20.3	36.1	722	78,413	75.5	40.9

Table 4. TABLE IV: Ablation studies where we demonstrate the advantage of Segmented Boxes and FRVO. We replace Segmented Boxes with regular Bounding Boxes (BBox) and compare its performance with DensePeds on the MOT benchmark. We also replace the FRVO model in turn with a constant velocity model (Const. Vel), the Social Forces model (SF)  [ 31 ] and the standard RVO model (RVO)  [ 6 ] and compare them on our dense crowd dataset. Bold is best. Arrows ( ↑ , ↓ ↑ ↓ \uparrow,\downarrow ) indicate the direction of better performance. Observation: Using FRVO improves the MOTA by 13.3% over the next best alternate. Using Segmented Boxes reduces the false negatives by 20.7%.

Detection	MT(%)$\uparrow$	ML(%)$\downarrow$	IDS$\downarrow$	FN$\downarrow$	MOTP(%)$\uparrow$	MOTA(%)$\uparrow$
BBox	14.0	44.7	313	34,716	76.4	17.6
DensePeds	19.0	32.7	429	27,499	75.6	20.0
Motion Model	MT(%)$\uparrow$	ML(%)$\downarrow$	IDS$\downarrow$	FN$\downarrow$	MOTP(%)$\uparrow$	MOTA(%)$\uparrow$
Const. Vel	0	98.0	33	82,467	64.2	67.1
SF	0	96.7	145	81,936	64.5	67.3
RVO	0	96.1	155	81,832	64.4	67.8
DensePeds	7.0	43.4	1,208	58,235	65.7	76.3

Equations

$$\begin{split}0\leq\Delta&\leq\sin^{-1}\textstyle\Big{(}\frac{2r}{\lVert\sigma\rVert}\Big{)}-\sin^{-1}\textstyle\Big{(}\frac{2r-\delta}{\lVert\sigma\rVert}\Big{)}\\ &\leq\sin^{-1}\textstyle\Big{(}\frac{2r}{\lVert\sigma\rVert}\sqrt{1-\Big{(}\frac{\delta-2r}{\lVert\sigma\rVert}\Big{)}^{2}}+\frac{\delta-2r}{\lVert\sigma\rVert}\sqrt{1-\Big{(}\frac{2r}{\lVert\sigma\rVert}\Big{)}^{2}}\Big{)}\\ \end{split}$$

$$\Psi_{t}=\big{[}x,v,v_{\textrm{pref}},{l},{w}\big{]},$$

$${FRVO}_{p_i}^{\tau }=\bigcup _{p_j\in {\mathcal{H}}_i}{VO}_{p_i|p_j}^{\tau }$$

$$v_{\text{new}}=\underset{v}{\mathrm{argmax}}\Vert v-v_{\text{pref}}\Vert$$

$$h_j^{\ast }=\underset{h_j}{\mathrm{argmin}}(d(f_{p_i},f_{h_j})|p_i\in \mathcal{P},h_j\in {\mathcal{H}}_i).$$

$$\zeta (i,j)=\frac{{\mathbb{B}}_{p_i}\cap {\mathbb{B}}_{h_j}}{{\mathbb{B}}_{p_i}\cup {\mathbb{B}}_{h_j}},h_j\in {\mathcal{H}}_i.$$

$$\mathcal{U}=\{{\mathbb{W}}_{h_j}({\mathbb{M}}_{h_j})|\mathbb{W}\in \mathcal{W},\mathbb{M}\in \mathcal{M},h_j\in \mathcal{H}\},$$

$$f_{p_i}^T(f_{h_j}^M-f_{h_j}^F)>0$$

$$d(f_{p_i},f_{h_j^{\ast }})>\lambda \iff (\alpha =\varphi )$$

$$\mathbb{P}(d(f_{p_i},f_{h_{j,p_i}^{\ast }}^M)>\lambda )<\mathbb{P}(d(f_{p_i},f_{h_{j,p_i}^{\ast }}^F)>\lambda )$$

$${\mathbb{P}}^M(\alpha =\varphi )<{\mathbb{P}}^F(\alpha =\varphi )$$

$$(\mu >\xi )\wedge (\alpha =\varphi )\iff {\mathcal{T}}_t\leftarrow \{\varphi \}$$

$${\mathbb{P}}^M({\mathcal{T}}_t\leftarrow \{\varphi \})<{\mathbb{P}}^F({\mathcal{T}}_t\leftarrow \{\varphi \})$$

$$FN=\sum _{t=1}^T\sum _{p_g\in \mathcal{G}}{\delta }_{{\mathcal{T}}_t}$$

Full text

I Introduction

Pedestrian tracking is the problem of maintaining the consistency in the temporal and spatial identity of a person in an image sequence or a crowd video. This is an important problem that helps us not only extract trajectory information from a crowd scene video but also helps us understand high-level pedestrian behaviors [1]. Many applications in robotics and computer vision such as action recognition and collision-free navigation and trajectory prediction [2] require tracking algorithms to work accurately in real time [3]. Furthermore, it is crucial to develop general-purpose algorithms that can handle front-facing cameras (used for robot navigation or autonomous driving) as well as elevated cameras (used for urban surveillance), especially in densely crowded areas such as airports, railway terminals, or shopping complexes.

Closely related to pedestrian tracking is pedestrian detection, which is the problem of detecting multiple individuals in each frame of a video. Pedestrian detection has received a lot of traction and gained significant progress in recent years. However, earlier work in pedestrian tracking [4, 5] did not include pedestrian detection. These tracking methods require manual, near-optimal initialization of each pedestrian’s state information in the first video frame. Further, sans-detection methods need to know the number of pedestrians in each frame apriori, so they do not handle cases in which new pedestrians enter the scene during the video. Tracking by detection overcomes these limitations by employing a detection framework to recognize pedestrians entering at any point of time during the video and automatically initialize their state-space information.

However, tracking pedestrians in dense crowd videos where there are 2 or more pedestrians per square meter remains a challenge for the tracking-by-detection literature. These videos suffer from severe occlusion, mainly due to the pedestrians walking extremely close to each other and frequently crossing each other’s paths. This makes it difficult to track each pedestrian across the video frames. Tracking-by-detection algorithms compute a bounding box around each pedestrian. Because of their proximity in dense crowds, the bounding boxes of nearby pedestrians overlap which affects the accuracy of tracking algorithms.

Main Contributions. Our main contributions in this work are threefold.

We present a pedestrian tracking algorithm called DensePeds that can efficiently track pedestrians in crowds with 2 or more pedestrians per square meter. We call such crowds “dense”. We introduce a new motion model called Frontal Reciprocal Velocity Obstacles (FRVO), which extends traditional RVO [6] to work with front- or elevated-view cameras. FRVO uses an elliptical approximation for each pedestrian and estimates pedestrian dynamics (position and velocity) in dense crowds by considering intermediate goals and collision avoidance constraints. 2. 2.

We combine FRVO with the state-of-the-art Mask R-CNN object detector [7] to compute sparse feature vectors. We show analytically that our sparse feature vector computation reduces the probability of the loss of pedestrian tracks (false negatives) in dense scenarios. We also show by experimentation that using sparse feature vectors makes our method outperform state-of-the-art tracking methods on dense crowd videos, without any additional requirement for training or optimization. 3. 3.

We make available a new dataset consisting of dense crowd videos. Available benchmark crowd videos are extremely sparse (less than 0.2 persons per square meter). Our dataset, on the other hand, presents a more challenging and realistic view of crowds in public areas in dense metropolitans.

On our dense crowd videos dataset, our method outperforms the state-of-the-art, by 2.6% and reduce false negatives by 11% on our dataset. We also validate the benefits of FRVO and sparse feature vectors through ablation experiments on our dataset. Lastly, we evaluate and compare the performance of DensePeds with state-of-the-art online methods on the MOT benchmark [8] for a comprehensive evaluation. The benefits of our method do not hold for sparsely crowded videos such as the ones in the MOT benchmark since FRVO is based on reciprocal velocities of two colliding pedestrians. Unsurprisingly, we do not outperform the state-of-the-art methods on the MOT benchmark since the MOT sequences do not contain many colliding pedestrians, thereby rendering FRVO ineffective. Nevertheless, our method still has the lowest false negatives among all the available methods. Its overall performance is state of the art on dense videos and is in the top 13% of all published methods on the MOT15 and top 20% on MOT16.

II Related Work

The most relevant prior work for our work is on object detection, pedestrian tracking, and motion models used in pedestrian tracking, which we present below.

II-A Object Detection

Early methods for object detection include HOG [9] and SIFT [10], which manually extract features from images. Inspired by AlexNet, [11] proposed R-CNN and its variants [12, 13] for optimizing the object detection problem by incorporating a selective search.

More recently, the prevalence of CNNs has led to the development of Mask R-CNN [7], which extends Faster R-CNN to include pixel-level segmentation. The use of Mask R-CNN in pedestrian tracking has been limited, although it has been used for other pedestrian-related problems such as pose estimation [14].

II-B Pedestrian Tracking

There have been considerable advances in object detection since the advent of to deep learning, which has led to substantial research in the intersection of deep learning and the tracking-by-detection paradigm [15, 16, 17, 18]. However, [18, 16] operate at less than one fps on a GPU and have low accuracy on standard benchmarks while [15] sacrifices accuracy to increase tracking speed. Finally, deep learning methods require expensive computation, often prohibiting real-time performance with inexpensive computing resources. Some recent methods [19, 20] achieve high accuracy but may require high-quality, heavily optimized detection features for good performance. For an up-to-date review of tracking-by-detection algorithms, we refer the reader to methods submitted to the MOT [8] benchmark.

II-C Motion Models in Pedestrian Tracking

Motion models are commonly used in pedestrian tracking algorithms to improve tracking accuracy [21, 22, 23, 24]. [22] presents a variation of MHT [25] and shows that it is at par with the state-of-the-art from the tracking-by-detection paradigm. [24] uses a motion model to combine fragmented pedestrian tracks caused by occlusion. These methods are based on linear constant velocity or acceleration models. Such linear models, however, cannot characterize pedestrian dynamics in dense crowds [26]. RVO [6] is a non-linear motion model that has been used for pedestrian tracking in dense videos to compute intermediate goal locations, but it only works with top-facing videos and circular pedestrian representations. RVO has been extended to tracking road-agents such as cars, buses, and two-wheelers, in addition to pedestrians, using a linear runtime motion model that considers both collision avoidance and pair-wise heterogeneous interactions between road-agents [27]. Other motion models used in pedestrian tracking are the Social Force model [28], LTA [29], and ATTR [30].

There are also many discrete motion models that represent each individual or pedestrian in a crowd as a particle (or as a 2D circle on a plane) to model the interactions. These include models based on repulsive forces [31] and velocity-based optimization algorithms [32],  [6]. More recent discrete approaches are based on short-term planning using a discrete approach [33] and cognitive models [34]. However, these methods are based on circular agent representation and do not work well for front-facing pedestrians in dense crowd videos as they are overly conservative in terms of motion prediction.

III FRVO: Local Trajectory Prediction

When walking in dense crowds, pedestrians keep changing their velocities frequently to avoid collisions with other pedestrians. Pedestrians also exhibit local interactions and other collision-avoidance behaviors such as side-stepping, shoulder-turning, and backpedaling [35]. As a result, prior motion models with constant velocity or constant acceleration assumptions do not accurately model crowded scenarios. Further, defining an accurate motion model gets even more challenging in front-view videos due to occlusion and proximity issues. For example, in top-view videos, one uses circular representations to model the shape of the heads of pedestrians. It is, generally, physically impossible for two individual’s heads to occlude or occupy the same space. In front facing crowd videos, however, occlusion is a common problem. In this section, we introduce our non-linear motion model, Frontal RVO (FRVO), which is designed to work with crowd videos captured using front- or elevated-view cameras.

III-A Notations

Table I lists the notations used in this paper. A few important points to keep in mind are:

• •

While $h_{j}$ denotes a detected pedestrian, $f_{h_{j}}$ denotes the feature vector extracted from the segmented box of $h_{j}$.

• •

$\mathcal{H}_{i}=\{h_{j}\ :\ |p_{i}-h_{j}|\leq\rho,p_{i}\in\mathcal{P}\}$

III-B Analytical Comparison with RVO

Our motion model is an extension of the RVO (Reciprocal Velocity Obstacle) [6] approach, which can accurately model the trajectories of pedestrians using collision avoidance constraints. However, RVO models pedestrians using circular shapes which result in false overlaps in front- and elevated-view camera videos (Figure 3). A false overlap is essentially a false positive wherein RVO would signal a collision while the actual scene would not contain a collision. We show here that false overlaps cause the accuracy of RVO to drop by an error margin of $\Delta$.

Definition III.1**.**

$\mathbf{\delta}$-overlap: Two circles each of radius $r$ are said to $\delta$-overlap if the length of the line joining their centers is equal to $2r-\delta$, where $0<\delta<2r$.

Now, consider the standard Velocity Obstacle formulation (the basis of RVO) [36] in Figure 2 (left). From the VO formulation, we have $R_{o}=2r$ and from III.1, $R_{i}=2r-\delta$. Let $\sigma$ denote the distance of $p_{j}$ from $p_{i}$. Then from simple geometry,

[TABLE]

$\Delta$ is upper-bounded by a function of $\delta$. Observe that the error bound increases for higher $\delta$. This is interpreted as follows: the more we increase the $\delta$-overlap, the transparent gray cone will correspondingly shrink, and the distance between the RVO generated velocity, and the ground truth velocity will increase.

We therefore require geometric representations for which $\delta\rightarrow 0$. We propose to model each pedestrian using an elliptical pedestrian representation, with the ellipse axes capturing the height of the pedestrian’s visible face and the shoulder length. We do not manually differentiate between the major and minor axes since ellipses can be tall or fat depending on the pedestrian orientation, and the particular axis to map either face length or shoulder width will vary. We observe that doing so produces an effective approximation. We only use the face and the shoulder in the FRVO formulation, because occlusion makes it difficult to observe other parts of the body in dense videos reliably.

III-C Computing Predicted Velocities

Each pedestrian $p_{i}$ is represented using the following 8-dimensional state vector:

[TABLE]

where $x$, $v$ , and $v_{\textrm{pref}}$ denote the current position of the pedestrian’s center of mass, the current velocity, and the preferred velocity, respectively. $l$ is the height of the pedestrian’s visible face, and $w$ captures the shoulder length. $v_{\textrm{pref}}$ is the velocity the pedestrian would have taken in the absence of obstacles or colliding pedestrians, computed using the standard RVO formulation.

We assume the pedestrians are oriented towards their direction of motion. For each frame and each pedestrian, we construct the half-plane constraints for each of its neighboring pedestrians and obstacles to predict its motion. We use velocity obstacles to compute the set of permitted velocities for a pedestrian. Given two pedestrians $p_{i}$ and $p_{j}$, the velocity obstacle of $p_{i}$ induced by $p_{j}$, denoted by $VO_{p_{i}|p_{j}}^{\tau}$ , constitutes the set of velocities for $p_{i}$ that would result in a collision with $p_{j}$ at some time before $\tau$. By definition, pedestrians $p_{i}$ and $p_{j}$ are guaranteed to be collision-free for at least time $\tau$, if ${{\vec{v}}_{p_{i}}}-{{\vec{v}}_{p_{j}}}\notin VO_{p_{i}|p_{j}}^{\tau}$. An pedestrian $p_{i}$ computes the velocity obstacle $VO_{p_{i}|p_{j}}^{\tau}$ for each of its neighboring pedestrians, $p_{j}$. The set of permitted velocities for a pedestrian $p_{i}$ is simply the convex region given by the intersection of the half-planes of the permitted velocities induced by all the neighboring pedestrians and obstacles. We denote this convex region for pedestrian $p_{i}$ and time horizon $\tau$ as ${FRVO}_{p_{i}}^{\tau}$. Thus,

[TABLE]

For each pedestrian $p_{i}$, we compute the new velocity $v_{\text{new}}$ from ${FRVO}_{p_{i}}^{\tau}$ that minimizes the deviation from its preferred velocity $v_{\text{pref}}$.

[TABLE]

such that $v\notin FRVO^{\tau}_{p_{i}}$.

In order to compute collision-free velocities, we need to compute the Minkowski Sums of ellipses. In practice, computing the exact Minkowski sums of ellipses is much more expensive as compared to those of circles. To overcome the complexity of exact Minkowski Sum computation, we compute conservative linear approximations of ellipses [35] and represent them as convex polygons. As a result, the collision avoidance problem reduces to linear programming. We use the predicted velocities to estimate the new position of a pedestrian in the next time-step.

IV DensePeds and Sparse Features

In this section, we describe our tracking algorithm, DensePeds, which is based on the tracking-by-detection paradigm. Then, we present our approach for sparse feature extraction and show how they reduce the number of false negatives during tracking.

IV-A DensePeds: Pedestrian Tracking Algorithm

Our algorithm uses the detected segmentation boxes given by Mask R-CNN to extract the sparse features. These features are matched using the Cosine metric $d(\cdot,\cdot)$ (ref. the last row of Table I) and the Hungarian algorithm [37] and are used to compute the position of each pedestrian during the next frame. We illustrate our approach in Figure 4. At current time $t$, given the ID labels of all pedestrians in the frame, we want to assign labels for pedestrians at time $t+1$. We start by using Mask R-CNN to implicitly perform pixel-wise pedestrian segmentation to generate segmented boxes, which are rectangular bounding boxes with background subtraction. Next, we predict the spatial coordinates of each person’s bounding box for the next time-step using FRVO, as shown in Figure 3 and Figure 3. This results in another set of bounding boxes for each pedestrian at time $t+1$.

We use these sets of boxes to compute appearance feature vectors, which are matched using an association algorithm [37]. This matching process is performed in two ways: the Cosine metric and the IoU overlap [38]. The Cosine metric is used to solve the following optimization problem to identify the detected pedestrian, $h_{j}$, that is most similar to $p_{i}$.

[TABLE]

The IoU overlap builds a cost matrix, $\zeta$ to measure the amount of overlap of each predicted bounding box with all nearby detection bounding box candidates. $\zeta(i,j)$ stores the IoU overlap of the bounding box of $\Psi_{t+1}$ with that of $h_{j}$ and is calculated as:

[TABLE]

Matching a detection to a predicted measurement with maximum overlap thus becomes a max weight matching problem and we solve it efficiently using the Hungarian algorithm [37]. The ID of the pedestrian at time $t$ is then assigned to the best matched pedestrian at time $t+1$.

IV-B Sparse Features

In dense crowds, the bounding boxes of nearby pedestrians have significant overlap. This overlap adds noise to the feature vector extracted from the bounding boxes. We address this problem by generating sparse feature vectors by subtracting the noisy background from the original bounding box to produce segmented boxes, which is described next.

IV-B1 Segmented Boxes Using Mask R-CNN

We use Mask R-CNN to perform pixel-wise person segmentation. In practice, Mask R-CNN essentially segments out the pedestrian from its bounding box, and thereby reduces the noise that occurs when the pedestrians are nearby.

Mask R-CNN generates a bounding box and its corresponding mask for each detected pedestrian in each frame. We create a white canvas and superimpose a pixel-wise segmented pedestrian onto the canvas using the mask. We perform detection at current time $t$ and the output consists of bounding boxes, masks, scores, and class IDs of pedestrians.

$\mathcal{B}=\{\mathbb{B}_{h_{j}}\ |\ \mathbb{B}=[\textrm{top left},m,n],h_{j}\in\mathcal{H}\}$ denotes the set of bounding boxes, where $\textrm{top left},m,n$ denote the top left corner, width, height of $\mathbb{B}_{j}$, respectively.

$\mathcal{M}=\{\mathbb{M}_{h_{j}}\ |\ h_{j}\in\mathcal{H}\}$ denotes the set of masks for each $h_{j}$, where each $\mathbb{M}_{h_{j}}$ is a $[m\times n]$ tensor of booleans.

Let $\mathcal{W}=\{\mathbb{W}_{h_{j}}(\cdot)\ |\ h_{j}\in\mathcal{H}\}$ be the set of white canvases where each canvas, $\mathbb{W}_{h_{j}}=[\mathbb{1}]_{m\times n}$, $w$ and $h$. Then,

[TABLE]

is the set of segmented boxes for each $h_{j}$ at time $t$. These segmented boxes are used by our real-time tracking algorithm shown in Figure 4.

The segmented boxes are input to the DeepSORT CNN [19] to compute binary feature vectors. Since a large portion of a segmented box contains zeros, these feature vectors are mostly sparse. We refer to these vectors as our sparse feature vectors. Our choice of the network parameters governs the size of the feature vectors.

IV-B2 Reduced Probability of Track Loss

We now show how sparse feature vectors generated from segmented boxes reduce the probability of the loss of pedestrian tracks (false negatives).

We define $\mathcal{T}_{t}=\{\Psi_{1:t}\}$ to be the set of positively identified states for $p_{i}$ until time $t$. We denote the time since the last update to a track ID as $\mu$. We denote the ID of $p_{i}$ as $\alpha$ and we represent the correct assignment of an ID to $p_{i}$ as $\Gamma(\alpha)$. The threshold for the Cosine metric is $\lambda\underset{\textrm{i.i.d.}}{\sim}\mathbb{U}[0,1]$. The threshold for the track age, i.e., the number of frames before which track is destroyed, is $\xi$. We denote the probability of an event that uses Mask R-CNN as the primary object detection algorithm with $\mathbb{P}^{M}(\cdot)$ and the probability of an event that uses a standard Faster R-CNN [13] as the primary object detection algorithm (i.e., outputs bounding boxes without boundary subtraction) with $\mathbb{P}^{F}(\cdot)$. Finally, $\mathcal{T}_{t}\leftarrow\{\phi\}$ represents the loss of $\mathcal{T}_{t}$ by occlusion.

We now state and prove the following lemma.

Lemma IV.1**.**

For every pair of feature vectors $(f^{M}_{h_{j}},f^{F}_{h_{j}})$ generated from a segmented box and a bounding box respectively, if $\lVert f^{M}_{h_{j}}\lVert_{0}>\lVert f^{F}_{h_{j}}\rVert_{0}$, then $\mathnormal{d}(f_{p_{i}},f^{M}_{h_{j}})<\mathnormal{d}(f_{h_{j}},f^{F}_{h_{j}})$ with probability $1-\frac{B}{A}$, where $A$ and $B$ are positive integers and $A>B$.

Proof.

Using the definition of the Cosine metric, the lemma reduces to proving the following,

[TABLE]

We pad both $f^{M}_{h_{j}}$ and $f^{F}_{h_{j}}$ such that $\lVert f^{M}_{h_{j}}\rVert_{0}>||f^{F}_{h_{j}}||_{0}$.

We reduce $f^{T}_{p_{i}}$, $f^{M}_{h_{j}}$, and $f^{F}_{h_{j}}$ to binary vectors, i.e., vectors composed of [math]s and $1$s. Let $\Delta f=f^{M}_{h_{j}}-f^{F}_{h_{j}}$. We denote the number of $1$s and $-1$s in $\Delta f$ as $A$ and $B$, respectively. Now, let $x$ and $y$ denote the $L_{0}$ norm of $f^{M}_{h_{j}}$ and $f^{F}_{h_{j}}$, respectively. From our padding procedure, we have $x>y$. Then, if $x=A$, and $y=B$, we trivially have $A>B$. But if $y>B$, then $A=x-(y-B)\implies A-B=x-y$. From $x>y$, it again follows that $A>B$. Thus, $x>y\implies A>B$.

Next, we define a $(1,1)$ coordinate in an ordered pair of vectors as the coordinate where both vectors contain $1$s. Similarly, a $(1,-1)$ coordinate in an ordered pair of vectors is the coordinate where the first vector contains $1$ and the second vector contains $-1$. Then, let $p_{a}$ and $p_{b}$ respectively denote the number of $(1,1)$ coordinates and $(1,-1)$ coordinates in the pair $(f^{T}_{p_{i}},\Delta f)$. By definition, we have $0<p_{a}<A$ and $0<p_{b}<B$. Thus, if we assume $p_{a}$ and $p_{b}$ to be uniformly distributed, it directly follows that $\mathbb{P}(p_{a}>p_{b})=1-\frac{B}{A}$.

∎

Based on Lemma IV.1, we finally prove the following proposition.

Proposition IV.1**.**

With probability $1-\frac{B}{A}$, sparse feature vectors extracted from segmented boxes decrease the loss of pedestrian tracks, thereby reducing the number of false negatives in comparison to regular bounding boxes.

Proof.

In our approach, we use Mask R-CNN for pedestrian detection, which outputs bounding boxes and their corresponding masks. We use the mask and bounding box pair to generate a segmented box (Section IV-B1). The correct assignment of an ID depends on successful feature matching between the predicted measurement feature and the optimal detection feature. In other words,

[TABLE]

Using Lemma IV.1 and the fact that $\lambda\underset{i.i.d.}{\sim}\mathbb{U}[0,1]$,

[TABLE]

Using Eq. 5, it directly follows that,

[TABLE]

In our approach, we set

[TABLE]

Using Eq. 6, it follows that,

[TABLE]

We define the total number of false negatives (FN) as

[TABLE]

where $p_{g}\in\mathcal{G}$ denotes a ground truth pedestrian in the set of all ground truth pedestrians at current time $t$ and $\delta_{z}=1$ for $z=0$ and [math] elsewhere. This is a variation of the Kronecker delta function. Using Eq. 7 and Eq. 8, we can say that fewer lost tracks ($\mathcal{T}_{t}\leftarrow\{\phi\}$) indicate a smaller number of false negatives.

∎

The upper bound, $\mathbb{P}^{F}(\mathcal{T}_{t})$, in Eq. 7 depends on the amount of padding done to $f_{p_{i}}$ and $f_{h_{j}}$. A general observed trend is that a higher amount of padding results in a larger upper bound in Eq. 7.

V Experimental Evaluation

V-A Datasets

We evaluate DensePeds on both our dense pedestrian crowd dataset and on the popular MOT [8] benchmark. MOT is a standard benchmark for testing the performance of multiple object tracking algorithms, and it subsumes older benchmarks such as KITTI [46] and PETS [47]. When evaluating on MOT, we use the popular MOT16 and MOT15 sequences. However, the main drawback of the sequences in the MOT benchmark is that they do not contain dense crowd videos that we address in this paper.

Our dataset consists of 8 dense crowd videos where the crowd density ranges from 2 to 2.7 pedestrians per square meter. By comparison, the densest crowd video in the MOT benchmark has around 0.2 persons per square meter. All videos in our dataset are shot at an elevated view using a stationary camera. Seven of the eight videos are shot 1080p resolution, and one video is at 480p resolution.

While we agree that the current size of our dataset is prohibitively small for training existing deep learning-based object detectors, we showcase the results of our method on this dataset to validate our claims and analyses. We plan to expand and release a benchmark version of our dataset in the recent future.

V-B Evaluation Metrics and Methods

V-B1 Metrics

We use a subset of the standard CLEAR MOT metrics [48] for evaluating the performance of our algorithm. Specifically, the metrics we use are:

Number of mostly tracked trajectories (MT). Correct ID association with a pedestrian across frames for at least 80% of its life span in a video. 2. 2.

Number of mostly lost trajectories (ML). Correct ID association with a pedestrian across frames for at most 20% of its life span in the video. 3. 3.

False Negatives (FN). Total number of false negatives in object detection (i.e., pedestrian tracks lost) over the video. 4. 4.

Identity Switches (IDSW). The total number of identity switches over pedestrians in the video. 5. 5.

MOT Precision (MOTP). The percentage misalignment between all the predicted bounding boxes and corresponding ground truth boxes over the video. 6. 6.

MOT Accuracy (MOTA). Overall tracking accuracy taking into account false positives (FP), false negatives (FN), and ID switches (IDSW). It is defined as $MOTA=1-\dfrac{FP+FN+IDS}{GT}$, where $GT$ is the sum of annotated pedestrians in all the frames in the video.

These are the most important metrics in CLEAR MOT for evaluating a tracker’s performance. We exclude metrics that evaluate the performance of detection since object detection is not a contribution of this paper. We have also excluded the number of false positives from our tables since it is already included in the definition of MOTA and is not the focus of this paper. On the other hand, we highlight the results of false negatives to validate the theoretical formulation of our algorithm in Section III-B.

V-B2 Methods

There is a large number of tracking-by-detection methods on the MOT benchmark, and it is not feasible to evaluate our method against all of them in the limited scope of this paper. Further, many methods submitted on the MOT benchmark server are unpublished. Thus, to keep our evaluations as fair and competitive as possible, we compare with state-of-the-art online trackers on the MOT16 as well as MOT15 benchmarks that are published. By “state-of-the-art”, we refer to methods that have an average rank greater or equal to our average rank.

For evaluation on our dataset, in addition to online state-of-the-art published methods, we also require their code. The only methods that meet all these criteria are MOTDT [39] and MDP [40].

V-C Results

All results generated by DensePeds are obtained without any training on tracking datasets. Thus, DensePeds does not suffer from over-fitting or dataset bias. All the methods that we compare with are trained on the MOT sequences.

V-C1 Our Dataset

Table II compares DensePeds with MOTDT [39] and MDP [40] on our dense crowd dataset. MOTDT is currently the best online published tracker on the MOT16 benchmark with available code, and MDP is the second-best method with available code. We used their off-the-shelf implementations, with weights pre-trained on the MOT benchmark, to compare with DensePeds.

We observe that DensePeds produces the lowest false negatives of all the methods. Thus, by Proposition IV.1 and Table II, DensePeds provably reduces the number of false negatives on dense crowd videos.

The low MT and high ML scores for the sequences are not an indicator of poor performance; rather they are a consequence of the strict definitions of the metrics. For example, in dense crowd videos, it is challenging to maintain a track for 80% of a pedestrian’s life span. DensePeds has the highest MT and lowest ML percentages of all the methods. This explains our high number of ID switches as a higher number of pedestrians being tracked would correspondingly increase the likelihood of ID switches. Most importantly, the MOTA of DensePeds is 2.9% more than that of MOTDT and 2.6% more than that of MDP. Roughly speaking, this is equivalent to an average rank difference of 19 from MOTDT and 17 from MDP on the MOT benchmark. We do not include the tracking speed metric (Hz) as shown in Table III since that is a metric produced exclusively by the MOT benchmark server.

V-C2 MOT Benchmark

Although our algorithm is primarily targeted for dense crowds, in the interest of thorough analysis, we also evaluate DensePeds on sparser crowds. We select a wide range of methods (as described in Section V-B2) to compare with DensePeds and highlight its relative advantages and disadvantages in Table III.

As expected, we do not outperform state of the art on the MOT sequences due to its sparse crowd sequences. We specifically point to our low MOTA scores which we attribute to our algorithm requiring a more accurate ground truth than the ones provided by the MOT benchmark. Our detection and tracking are highly sensitive — they can track pedestrians that are too distant to be manually labeled. This erroneously leads to a higher count of false positives and reduces the MOTA. This was observed to be true for the methods we compared with as well. Consequently, we exclude FP from the calculation of MOTA for all methods in the interest of fair evaluation.

However, following Proposition IV.1, our method achieves the lowest number of false negatives among all the methods on the MOT benchmark as well. In terms of runtime, we are approximately 4.5$\times$ faster than the state-of-the-art methods on an NVIDIA Titan Xp GPU.

V-C3 Ablation Experiments

In Table IV, we validate the benefits of sparse feature vectors (obtained via segmented boxes) in detection and FRVO in tracking in dense crowds with ablation experiments.

First, we replace segmented boxes with regular bounding boxes (boxes without background subtraction), and compare its performance with DensePeds on the MOT benchmark. In accordance with Proposition IV.1, DensePeds reduces the number of false negatives by 20.7% on relative.

Next, we replace FRVO in turn with a constant velocity motion model [19], the Social Forces model [31], and the standard RVO model [6]; all without changing the rest of the DensePeds algorithm. All of these models fail to track pedestrians in dense crowds (ML nearly 100%). Note that a consequence of failing to track pedestrians is the unusually low number of ID switches that we observe for these methods. This, in turn, leads to their comparatively high MOTA, despite being mostly unable to track pedestrians. Using FRVO, we decrease ML by 55% on relative. At the same time, our MOTA improves by a significant 12% on relative and 8.5% on absolute.