Deep Network Flow for Multi-Object Tracking

TL;DR

This paper introduces a method to learn features for network-flow-based multi-object tracking by making the data association process differentiable, enabling end-to-end training and outperforming traditional hand-crafted costs.

Contribution

It demonstrates that network flow costs in multi-object tracking can be learned via backpropagation, improving accuracy over hand-crafted methods.

Findings

Learned cost functions outperform hand-crafted costs in all tested scenarios.

End-to-end training simplifies the integration of multiple input sources.

The approach is applicable to various data association problems in computer vision.

Abstract

Data association problems are an important component of many computer vision applications, with multi-object tracking being one of the most prominent examples. A typical approach to data association involves finding a graph matching or network flow that minimizes a sum of pairwise association costs, which are often either hand-crafted or learned as linear functions of fixed features. In this work, we demonstrate that it is possible to learn features for network-flow-based data association via backpropagation, by expressing the optimum of a smoothed network flow problem as a differentiable function of the pairwise association costs. We apply this approach to multi-object tracking with a network flow formulation. Our experiments demonstrate that we are able to successfully learn all cost functions for the association problem in an end-to-end fashion, which outperform hand-crafted costs in all settings. The integration and combination of various sources of inputs becomes easy and the cost functions can be learned entirely from data, alleviating tedious hand-designing of costs.

Tables

Table 1. (a)

	MOTA	REC	PREC	MT	IDS	FRAG
Crafted [29]	73.64	83.54	92.99	58.73	121	459
Crafted-ours	73.75	83.92	92.65	59.44	89	431
Linear	73.51	83.47	92.99	59.08	132	430
MLP 1	74.09	83.93	92.87	59.61	70	371
MLP 2	74.19	84.07	92.85	59.96	70	376

Table 2. Table 4 : Results on KITTI-Tracking  [ 17 ] from 11/04/16.

Method	MOTA	MOTP	MT	ML	IDS	FRAG
[30]	60.84	78.55	53.81	7.93	191	966
[10]	69.73	79.46	56.25	12.96	36	225
[34]	55.49	78.85	36.74	14.02	323	984
[50]	66.35	77.80	55.95	8.23	63	558
Ours	67.36	78.79	53.81	9.45	65	574

Equations

$${\mathbf{x}}^{\ast }=$$

$$\mathbf{Ax}\le \mathbf{b},\mathbf{Cx}=0,$$

$$\underset{\Theta }{\mathrm{argmin}}$$

$${\mathbf{x}}^{\ast }=\underset{\mathbf{x}}{\mathrm{argmin}}\mathbf{c}(\mathbf{f},\Theta {)}^{\top }\mathbf{x}$$

$$\mathbf{Ax}\le \mathbf{b},\mathbf{Cx}=0,$$

$${\mathbf{x}}^{\ast }=$$

$$\mathbf{Cx}=0,$$

$$\underset{\mathbf{z}}{\mathrm{argmin}}t\cdot \mathbf{c}(\mathbf{f},\Theta {)}^{\top }\mathbf{x}(\mathbf{z})+P(\mathbf{x}(\mathbf{z})),$$

$$\underset{\mathbf{\Theta }}{\mathrm{argmin}}$$

$${\mathbf{z}}^{\ast }=\underset{\mathbf{z}}{\mathrm{argmin}}t\cdot \mathbf{c}(\mathbf{f},\mathbf{\Theta }{)}^{\top }\mathbf{x}(\mathbf{z})+P(\mathbf{x}(\mathbf{z})),$$

$$\frac{\partial \mathcal{L}}{\mathbf{c}}$$

$$0$$

$$=t\cdot \frac{\partial \mathbf{x}}{\partial \mathbf{z}}\mathbf{c}+\frac{\partial \mathbf{x}}{\partial \mathbf{z}}\frac{\partial P}{\partial \mathbf{x}}=t\cdot {\mathbf{B}}^{\top }\mathbf{c}+{\mathbf{B}}^{\top }\frac{\partial P}{\partial \mathbf{x}}$$

$$0$$

$$=t\cdot \mathbf{B}+\frac{\partial \mathbf{z}}{\partial \mathbf{c}}\frac{\partial \mathbf{x}}{\partial \mathbf{z}}\frac{{\partial }^{2}P}{\partial {\mathbf{x}}^2}\mathbf{B}=t\cdot \mathbf{B}+\frac{\partial \mathbf{z}}{\partial \mathbf{c}}{\mathbf{B}}^{\top }\frac{{\partial }^{2}P}{\partial {\mathbf{x}}^2}\mathbf{B}$$

$$\frac{\partial \mathbf{z}}{\partial \mathbf{c}}=-t\cdot \mathbf{B}{\left[{\mathbf{B}}^{\top }\frac{{\partial }^{2}P}{\partial {\mathbf{x}}^2}\mathbf{B}\right]}^{-1}.$$

$$\frac{\partial \mathcal{L}}{\mathbf{c}}=-t\cdot \mathbf{B}{\left[{\mathbf{B}}^{\top }\frac{{\partial }^{2}P}{\partial {\mathbf{x}}^2}\mathbf{B}\right]}^{-1}{\mathbf{B}}^{\top }\frac{\partial \mathcal{L}}{\partial \mathbf{x}}.$$

$$\frac{{\partial }^{2}P}{\partial {\mathbf{x}}^2}=\frac{{\partial }^{2}P}{\partial \mathbf{x}\partial {\mathbf{x}}^{\top }}=\sum _{i=1}^{2M}\frac{1}{{\left(b_i-{\mathbf{a}}_i^{\top }\mathbf{x}\right)}^2}\cdot {\mathbf{a}}_i{\mathbf{a}}_i^{\top }.$$

$$\mathcal{L}\left({\mathbf{x}}^{\ast },{\mathbf{x}}^{\text{gt}}\right)=$$

$$+\sum _{i,j\in \mathcal{E}}{\omega }_{ij}(x_{i,j}^{\text{link},\ast }-x_{i,j}^{\text{gt}}{)}^2,$$

$$c_{i,j}^{\text{link}}=-\log \text{E}\left(\frac{\Vert \mathbf{b}({\mathbf{d}}_i)-\mathbf{b}({\mathbf{d}}_j)\Vert }{{\Delta }_t},V_{\text{max}}\right)-\log (B^{{\Delta }_t-1}),$$

$$c_{i,j}^{\text{link}}=(1-\text{IoU}(\mathbf{b}({\mathbf{d}}_i),\mathbf{b}({\mathbf{d}}_j)))+\beta \cdot ({\Delta }_t-1),$$

$$c_{i,j}^{\text{link}}=(1-\text{IoU}(\mathbf{b}({\mathbf{d}}_i),\mathbf{b}({\mathbf{d}}_j)))+\beta \cdot ({\Delta }_t-1)+\gamma \cdot (1-{\hat{s}}_{\text{A}}),$$

$$c_i^{\text{det}}={\mathbf{c}}_{\text{conf}}(p({\mathbf{d}}_i);{\mathbf{\Theta }}_{\text{conf}})+{\mathbf{c}}_{\text{Au}}(I_i;{\mathbf{\Theta }}_{\text{Au}}),$$

Full text

1 Introduction

Multi-object tracking (MOT) is the task of predicting the trajectories of all object instances in a video sequence. MOT is challenging due to occlusions, fast moving objects or moving camera platforms, but it is an essential module in many applications like action recognition, surveillance or autonomous driving. The currently predominant approach to MOT is tracking-by-detection [3, 7, 10, 15, 26, 33, 41], where, in a first step, object detectors like [16, 43, 51] provide potential locations of the objects of interest in the form of bounding boxes. Then, the task of multi-object tracking translates into a data association problem where the bounding boxes are assigned to trajectories that describe the path of individual object instances over time.

Bipartite graph matching [25, 35] is often employed in on-line approaches to assign bounding boxes in the current frame to existing trajectories [22, 37, 38, 52]. Off-line methods can be elegantly formulated in a network flow framework to solve the association problem including birth and death of trajectories [27, 29, 54]. Section 2 gives more examples. All these association problems can be solved in a linear programming (LP) framework, where the constraints are given by the problem. The interplay of all variables in the LP, and consequently their costs, determines the success of the tracking approach. Hence, designing good cost functions is crucial. Although cost functions are hand-crafted in most prior work, there exist approaches for learning costs from data. However, they either do not treat the problem as a whole and only optimize parts of the costs [27, 31, 52, 54] or are limited to linear cost functions [49, 50].

We propose a novel formulation that allows for learning arbitrary parameterized cost functions for all variables of the association problem in an end-to-end fashion, i.e., from input data to the solution of the LP. By smoothing the LP, bi-level optimization [6, 13] enables learning of all the parameters of the cost functions such as to minimize a loss that is defined on the solution of the association problem, see Section 3.2. The main benefit of this formulation is its flexibility, general applicability to many problems and the avoidance of tedious hand-crafting of cost functions. Our approach is not limited to log-linear models (c.f., [49]) but can take full advantage of any differentiable parameterized function, e.g., neural networks, to predict costs. Indeed, our formulation can be integrated into any deep learning framework as one particular layer that solves a linear program in the forward pass and back-propagates gradients w.r.t. the costs through its solution (see Figure 2).

While our approach is general and can be used for many association problems, we explore its use for multi-object tracking with a network flow formulation (see Sections 3.1 and 3.4). We empirically demonstrate on public data sets [17, 28, 32] that: (i) Our approach enables end-to-end learning of cost functions for the network flow problem. (ii) Integrating different types of input sources like bounding box information, temporal differences, appearance and motion features becomes easy and all model parameters can be learned jointly. (iii) The end-to-end learned cost functions outperform hand-crafted functions without the need to hand-tune parameters. (iv) We achieve encouraging results with appearance features, which suggests potential benefits from end-to-end integration of deep object detection and tracking, as enabled by our formulation.

2 Related Work

Association problems in MOT:

Recent works on multi-object tracking (MOT) mostly follow the tracking-by-detection paradigm [3, 7, 10, 15, 26, 33, 41], where objects are first detected in each frame and then associated over time to form trajectories for each object instance. On-line methods like [8, 11, 15, 39, 41] associate detections of the incoming frame immediately to existing trajectories and are thus appropriate for real-time applications111In this context, real-time refers to a causal system.. Trajectories are typically treated as state-space models like Kalman [21] or particle filters [18]. The association to bounding boxes in the current frame is often formulated as bipartite graph matching and solved via the Hungarian algorithm [25, 35]. While on-line methods only have access to the past and current observations, off-line (or batch) approaches [3, 9, 20, 1, 40, 54] also consider future frames or even the whole sequence at once. Although not applicable for real-time applications, the advantage of batch methods is the temporal context allowing for more robust and non-greedy predictions. An elegant solution to assign trajectories to detections is the network flow formulation [54] (see Section 3.1 for details). Both of these association models can be formulated as linear program.

Cost functions:

Independent of the type of association model, a proper choice of the cost function is crucial for good tracking performance. Many works rely on carefully designed but hand-crafted functions. For instance, [29, 33, 41] only rely on detection confidences and spatial (i.e., bounding box differences) and temporal distances. Zhang et al. [54] and Zamir et al. [53] include appearance information via color histograms. Other works explicitly learn affinity metrics, which are then used in their tracking formulation. For instance, Li et al. [31] build upon a hierarchical association approach where increasingly longer tracklets are combined into trajectories. Affinities between tracklets are learned via a boosting formulation from various hand-crafted inputs including length of trajectories and color histograms. This approach is extended in [26] by learning affinities on-line for each sequence. Similarly, Bae and Yoon [2] learn affinities on-line with a variant of linear discriminant analysis. Song et al. [47] train appearance models on-line for individual trajectories when they are isolated, which can then be used to disambiguate from other trajectories in difficult situations like occlusions or interactions. Leal-Taixé et al. [27] train a Siamese neural network to compare the appearance (raw RGB patches) of two detections and combine this with spatial and temporal differences in a boosting framework. These pair-wise costs are used in a network flow formulation similar to [29]. In contrast to our approach, none of these methods consider the actual inference model during the learning phase but rely on surrogate loss functions for parts of the tracking costs.

Integrating inference into learning:

Similar to our approach, there have been recent works that also include the full inference model in the training phase. In particular, structured SVMs [48] have recently been used in the tracking context to learn costs for bipartite graph matching in an on-line tracker [23], a divide-and-conquer tracking strategy [46] and a joint graphical model for activity recognition and tracking [12]. In a similar fashion, [49] present a formulation to jointly learn all costs in a network flow graph with a structured SVM, which is the closest work to ours. It shows that properly learning cost functions for a relatively simple model can compete with complex tracking approaches. However, the employed structured SVM limits the cost functions to a linear parameterization. In contrast, our approach relies on bi-level optimization [6, 13] and is more flexible, allowing for non-linear (differentiable) cost functions like neural networks. Bi-level optimization has also been used recently to learn costs of graphical models, e.g., for segmentation [42] or depth map restoration [44, 45].

3 Deep Network Flows for Tracking

We demonstrate our end-to-end formulation for association problems with the example of network flows for multi-object tracking. In particular, we consider a tracking-by-detection framework, where potential detections $\mathbf{d}$ in every frame $t$ of a video sequence are given. Each detection consists of a bounding box $\mathbf{b}(\mathbf{d})$ describing the spatial location, a detection probability $p(\mathbf{d})$ and a frame number $t(\mathbf{d})$. For each detection, the tracking algorithm needs to either associate it with an object trajectory $\mathcal{T}_{k}$ or reject it. A trajectory is defined as a set of detections belonging to the same object, i.e., $\mathcal{T}_{k}=\{\mathbf{d}_{k}^{1},\ldots,\mathbf{d}_{k}^{N_{k}}\}$, where $N_{k}$ defines the size of the trajectory. Only bounding boxes from different frames can belong to the same trajectory. The number of trajectories $\left|\mathcal{T}\right|$ is unknown and needs to be inferred as well.

In this work, we focus on the network flow formulation from Zhang et al. [54] to solve the association problem. It is a popular choice [27, 29, 30, 49] that works well in practice and can be solved via linear programming (LP). Note that bipartite graph matching, which is typically used for on-line trackers, can also be formulated as a network flow, making our learning approach equally applicable.

3.1 Network Flow Formulation

We present the formulation of the directed network flow graph with an example illustrated in Figure 1. Each detection $\mathbf{d}_{i}$ is represented with two nodes connected by an edge (red). This edge is assigned the flow variable $x^{\textrm{det}}_{i}$. To be able to associate two detections, meaning they belong to the same trajectory $\mathcal{T}$, directed edges (blue) from all $\mathbf{d}_{i}$ (second node) to all $\mathbf{d}_{j}$ (first node) are added to the graph if $t(\mathbf{d}_{i})<t(\mathbf{d}_{j})$ and $|t(\mathbf{d}_{i})-t(\mathbf{d}_{j})|<\tau_{t}$. Each of these edges is assigned a flow variable $x^{\textrm{link}}_{i,j}$. Having edges over multiple frames allows for handling occlusions or missed detections. To reduce the size of the graph, we drop edges between detections that are spatially far apart. This choice relies on the smoothness assumption of objects in videos and does not hurt performance but reduces inference time. In order to handle birth and death of trajectories, two special nodes are added to the graph. A source node (S) is connected with the first node of each detection $\mathbf{d}_{i}$ with an edge (black) that is assigned the flow variable $x^{\textrm{in}}_{i}$. Similarly, the second node of each detection is connected with a sink node (T) and the corresponding edge (black) is assigned the variable $x^{\textrm{out}}_{i}$.

Each variable in the graph is associated with a cost. For each of the four variable types we define the corresponding cost, i.e., $c^{\textrm{in}}$, $c^{\textrm{out}}$, $c^{\textrm{det}}$ and $c^{\textrm{link}}$. For ease of explanation later, we differentiate between unary costs $\mathbf{c}^{\textrm{U}}$ ($c^{\textrm{in}}$, $c^{\textrm{out}}$ and $c^{\textrm{det}}$) and pairwise costs $c^{\textrm{P}}$ ($c^{\textrm{link}}$). Finding the globally optimal minimum cost flow can be formulated as the linear program

[TABLE]

where $\mathbf{x}\in\mathbb{R}^{M}$ and $\mathbf{c}\in\mathbb{R}^{M}$ are the concatenations of all flow variables and costs, respectively, and $M$ is the problem dimension. Note that we already relaxed the actual integer constraint on $\mathbf{x}$ with box constraints $0\leq\mathbf{x}\leq 1$, modeled by $\mathbf{A}=[\mathbf{I},-\mathbf{I}]^{\top}\in\mathbb{R}^{2M\times M}$ and $\mathbf{b}=[\mathbf{1},\mathbf{0}]^{\top}\in\mathbb{R}^{2M}$ in (1). The flow conservation constraints, $x^{\textrm{in}}_{i}+\sum_{j}x^{\textrm{link}}_{ji}=x^{\textrm{det}}_{i}$ and $x^{\textrm{out}}_{i}+\sum_{j}x^{\textrm{link}}_{ij}=x^{\textrm{det}}_{i}\;\forall i$, are modeled with $\mathbf{C}\in\mathbb{R}^{2K\times M}$, where $K$ is the number of detections. The thick dashed lines in Figure 1 illustrate $\mathbf{x}^{*}$.

The most crucial part in this formulation is to find proper costs $\mathbf{c}$ that model the interplay between birth, existence, death and association of detections. The final tracking result mainly depends on the choice of $\mathbf{c}$.

3.2 End-to-end Learning of Cost Functions

The main contribution of this paper is a flexible framework to learn functions that predict the costs of all variables in the network flow graph. Learning can be done end-to-end, i.e., from the input data all the way to the solution of the network flow problem. To do so, we replace the constant costs $\mathbf{c}$ in Equation (1) with parameterized cost functions $\mathbf{c}(\mathbf{f},\mathbf{\Theta})$, where $\mathbf{\Theta}$ are the parameters to be learned and $\mathbf{f}$ is the input data. For the task of MOT, the input data typically are bounding boxes, detection scores, images features, or more specialized and effective features like ALFD [10].

Given a set of ground truth network flow solutions $\mathbf{x}^{\textrm{gt}}$ of a tracking sequence (we show how to define ground truth in Section 3.3) and the corresponding input data $\mathbf{f}$, we want to learn the parameters $\mathbf{\Theta}$ such that the network flow solution minimizes some loss function. This can be formulated as the bi-level optimization problem

[TABLE]

which tries to minimize the loss function $\mathcal{L}$ (upper level problem) w.r.t. the solution of another optimization problem (lower level problem), which is the network flow in our case, i.e., the inference of the tracker. To compute gradients of the loss function w.r.t. the parameters $\mathbf{\Theta}$ we require a smooth lower level problem. The box constraints, however, render it non-smooth.

3.2.1 Smoothing the lower level problem

The box constraints in (1) and (2) can be approximated via log-barriers [5]. The inference problem then becomes

[TABLE]

where $t$ is a temperature parameter (defining the accuracy of the approximation) and $\mathbf{a}_{i}^{\top}$ are rows of $\mathbf{A}$. Moreover, we can get rid of the linear equality constraints with a change of basis $\mathbf{x}=\mathbf{x}(\mathbf{z})=\mathbf{x}_{0}+\mathbf{B}\mathbf{z}$, where $\mathbf{C}\mathbf{x}_{0}=\mathbf{0}$ and $\mathbf{B}=\mathcal{N}\left(\mathbf{C}\right)$, i.e., the null space of $\mathbf{C}$, making our objective unconstrained in $\mathbf{z}$ ($\mathbf{C}\mathbf{x}=\mathbf{C}\mathbf{x}_{0}+\mathbf{C}\mathbf{B}\mathbf{z}=\mathbf{C}\mathbf{x}_{0}=\mathbf{0}=\textrm{True}\;\forall\mathbf{z}$). This results in the following unconstrained and smooth lower level problem

[TABLE]

where $P(\mathbf{x})=-\sum_{i=1}^{2M}\log(b_{i}-\mathbf{a}_{i}^{\top}\mathbf{x})$.

3.2.2 Gradients with respect to costs

Given the smoothed lower level problem (4), we can define the final learning objective as

[TABLE]

which is now well-defined. We are interested in computing the gradient of the loss $\mathcal{L}$ w.r.t. the parameters $\mathbf{\Theta}$ of our cost function $\mathbf{c}(\cdot,\mathbf{\Theta})$. It is sufficient to show $\frac{\partial\mathcal{L}}{\partial\mathbf{c}}$, as gradients for the parameters $\mathbf{\Theta}$ can be obtained via the chain rule assuming $\mathbf{c}(\cdot;\mathbf{\Theta})$ is differentiable w.r.t. $\mathbf{\Theta}$.

The basic idea for computing gradients of problem (5) is to make use of implicit differentiation on the optimality condition of the lower level problem. For an uncluttered notation, we drop all dependencies of functions in the following. We define the desired gradient via chain rule as

[TABLE]

We assume the loss function $\mathcal{L}$ to be differentiable w.r.t. $\mathbf{x}$. To compute $\frac{\partial\mathbf{z}^{*}}{\partial\mathbf{c}}$, we use the optimality condition of (4)

[TABLE]

and differentiate w.r.t. $\mathbf{c}$, which gives

[TABLE]

and which can be rearranged to

[TABLE]

The final derivative can then be written as

[TABLE]

To fully define (10), we provide the second derivative of $P$ w.r.t. $\mathbf{x}$, which is given as

[TABLE]

In the supplemental material we show that (10) is equivalent to a generic solution provided in [36] and that $\mathbf{B}^{\top}\frac{\partial^{2}P}{\partial\mathbf{x}^{2}}\mathbf{B}$ is always invertible.

3.2.3 Discussion

Training requires to solve the smoothed linear program (4), which can be done with any convex solver. This is essentially one step in a path-following method with a fixed temperature $t$. As suggested in [5], we set $t=\frac{M}{\epsilon}$, where $\epsilon$ is a hyper-parameter defining the approximation accuracy of the log barriers. We tried different values for $\epsilon$ and also an annealing scheme, but the results seem insensitive to this choice. We found $\epsilon=0.1$ to work well in practice.

It is also important to note that our formulation is not limited to the task of MOT. It can be employed for any application where it is desirable to learn costs functions from data for an association problem, or, more generally, for a linear program with the assumptions given in Section 3.2.1. Our formulation can also be interpreted as one particular layer in a neural network that solves a linear program. The analogy between solving the smoothed linear program (4) and computing the gradients (10) with the forward and backward pass of a layer in a neural network is illustrated in Figure 2.

3.3 Defining ground truth and the loss function

To learn the parameters $\mathbf{\Theta}$ of the cost functions we need to compare the LP solution $\mathbf{x}^{*}$ with the ground truth solution $\mathbf{x}^{\textrm{gt}}$ in a loss function $\mathcal{L}$. Basically, $\mathbf{x}^{\textrm{gt}}$ defines which edges in the network flow graph should be active ($x^{\textrm{gt}}_{i}=1$) and inactive ($x^{\textrm{gt}}_{i}=0$). Training data needs to contain the ground truth bounding boxes (with target identities) and the detection bounding boxes. The detections define the structure of the network flow graph (see Section 3.1).

To generate $\mathbf{x}^{\textrm{gt}}$, we first match each detection with ground truth boxes in each frame individually. Similar to the evaluation of object detectors, we match the highest scoring detection having an intersection-over-union overlap larger $0.5$ to each ground truth bounding box. This divides the set of detection into true and false positives and already defines the ground truth for $x^{\textrm{det}}$. In order to provide ground truth for associations between detections, i.e., $x^{\textrm{link}}$, we iterate the frames sequentially and investigate all edges pointing forward in time for each detection. We activate the edge that points to the closest true positive detection in time, which has the same target identity. All other $x^{\textrm{link}}$ edges are set to [math]. After all ground truth trajectories are identified, it is straightforward to set the ground truth of $x^{\textrm{in}}$ and $x^{\textrm{out}}$.

As already pointed out in [50], there exist different types of links that should be treated differently in the loss function. There are edges $x^{\textrm{link}}$ between two false positives (FP-FP), between true and false positives (TP-FP), and between two true positives with the same (TP-TP+) or a different (TP-TP-) identity. For (TP-TP+) links, we also differentiate between the shortest links for the trajectory and links that are longer (TP-TP+Far). Edges associated with a single detection ($x^{\textrm{in}}$, $x^{\textrm{det}}$ and $x^{\textrm{out}}$) are either true (TP) or false positives (FP). Figure 3 illustrates all these cases. To trade-off the importance between these types, we define the following weighted loss function

[TABLE]

where $\mathcal{E}$ is the set of all edges between detections $i$ and $j$. Note that the weights can be adjusted for each variable separately. The default value for the weights is $1$, but we can adjust them to incorporate three intuitions about the loss. (i) Ambiguous edges: Detections of an (FP-FP) link may describe a consistently tracked but wrong object. Also, detections of a (TP-TP+Far) link are obviously very similar. In both cases the ground truth variable is still inactive. It may hurt the learning procedure if a wrong prediction is penalized too much for these cases. Thus, we can set $\omega_{i,j}=\omega_{\textrm{amb}}<1$. (ii) To influence the trade-off between precision and recall, we define the weight $\omega_{\textrm{pr}}$ for all edges involving a true positive detection. Increasing $\omega_{\textrm{pr}}$ favors recall. (iii) To emphasize associations, we additionally weight all $x^{\textrm{link}}$ variables with $\omega_{\textrm{link}}$. If multiple of these cases are true for a single variable, we multiply the weights.

Finally, we note that [50] uses a different weighting scheme and an $\ell_{1}$ loss. We compare this definition with various weightings of our loss function in Section 4.3.

3.4 Tracking model

After the training phase, the above described network flow formulation can be readily applied for tracking. One option is to batch process whole sequences at once, which, however, does not scale to long sequences. Lenz et al. [30] present a sophisticated approximation with bounded memory and computation costs. As we focus on the learning phase in this paper, we opt for a simpler approach, which empirically gives similar results to batch processing but does not come with guarantees as in [30].

We use a temporal sliding window of length $W$ that breaks a video sequence into chunks. We solve the LP problem for the frames inside the window, move it by $\Delta$ frames and solve the new LP problem, where $0<\Delta<W$ ensures a minimal overlap of the two solutions. Each solution contains a separate set of trajectories, which we associate with bipartite graph matching to carry the object identity information over time. The matching cost for each pair of trajectories is inversely proportional to the number of detections they share. Unmatched trajectories get new identities.

In practice, we use maximal overlap, i.e., $\Delta=1$, to ensure stable associations of trajectories between two LP solutions. For each window, we output the detections of the middle frame, i.e., looking $\frac{W}{2}$ frames into future and past, similar to [10]. Note that using detections from the latest frame as output enables on-line processing.

4 Experiments

To evaluate the proposed tracking algorithm we use the publicly available benchmarks KITTI tracking [17], MOT15 [28] and MOT16 [32]. The data sets provide training sets of $21$, $11$ and $7$ sequences, respectively, which are fully annotated. As suggested in [17, 28, 32], we do a ($4$-fold) cross validation for all our experiments, except for the benchmark results in Section 4.4.

To assess the performance of the tracking algorithms we rely on standard MOT metrics, CLEAR MOT [4] and MT/PT/ML [31], which are also used by both benchmarks [17, 28]. This set of metrics measures recall and precision, both on a detection and trajectory level, counts the number of identity switches and fragmentations of trajectories and also provides an overall tracking accuracy (MOTA).

4.1 Learned versus hand-crafted cost functions

The main contribution of this paper is a novel way to automatically learn parameterized cost functions for a network flow based tracking model from data. We illustrate the efficacy of the learned cost functions by comparing them with two standard choices for hand-crafted costs. First, we follow [29] and define $c^{\textrm{det}}_{i}=\log(1-p(\mathbf{d}_{i}))$, where $p(\mathbf{d}_{i})$ is the detection probability, and

[TABLE]

where $\textrm{E}(V_{t},V_{\textrm{max}})=\frac{1}{2}+\frac{1}{2}\textrm{erf}(\frac{-V_{t}+0.5\cdot V_{\textrm{max}}}{0.25\cdot V_{\textrm{max}}})$ with $\textrm{erf}(\cdot)$ being the Gauss error function and $\Delta_{t}$ is the frame difference between $i$ and $j$. While [29] defines a slightly different network flow graph, we keep the graph definition the same (see Section 3.1) for all methods to ensure a fair comparison of the costs. Second, we hand-craft our own cost function and define $c^{\textrm{det}}_{i}=\alpha\cdot p(\mathbf{d}_{i})$ as well as

[TABLE]

where $\textrm{IoU}(\cdot,\cdot)$ is the intersection over union. We tune all parameters, i.e., $c^{\textrm{in}}_{i}=c^{\textrm{out}}_{i}=C$ (we did not observe any benefit when choosing these parameters separately), $B$, $V_{\textrm{max}}$, $\alpha$ and $\beta$, with grid search to maximize MOTA while balancing recall. Note that the exponential growth of the search space w.r.t. the number of parameters makes grid search infeasible at some point.

With the same source of input information, i.e., bounding boxes $\mathbf{b}(\mathbf{d})$ and detection confidences $p(\mathbf{d})$, we train various types of parameterized functions with the algorithm proposed in Section 3.2. For unary costs, we use the same parameterization as for the hand-crafted model, i.e., constants for $c^{\textrm{in}}$ and $c^{\textrm{out}}$ and a linear model for $c^{\textrm{det}}$. However, for the pair-wise costs, we evaluate a linear model, a one-layer MLP with $64$ hidden neurons and a two-layer MLP with $32$ hidden neurons in both layers. The input feature $\mathbf{f}$ is the difference between the two bounding boxes, their detection confidences, the normalized time difference, as well as the IoU value. We train all three models for $50$k iterations using ADAM [24] with a learning rate of $10^{-4}$, which we decrease by a factor of $10$ every $20$k iterations.