$\ell_1$-minimization method for link flow correction

TL;DR

This paper introduces an $ ext{l}_1$-minimization approach for correcting inconsistent link flow data in road networks, effectively identifying and fixing corrupted sensors under certain recoverability conditions, even with measurement noise.

Contribution

It develops a novel $ ext{l}_1$-minimization method with a recoverability condition to accurately correct corrupted traffic flow data, including an analytical framework and algorithm for robustness.

Findings

Method accurately corrects corrupted link flows in synthetic and real data.

Recoverability condition ensures robustness to sensor miscounts.

Provides bounds on correction errors under measurement noise.

Abstract

A computational method, based on $\ell_1$-minimization, is proposed for the problem of link flow correction, when the available traffic flow data on many links in a road network are inconsistent with respect to the flow conservation law. Without extra information, the problem is generally ill-posed when a large portion of the link sensors are unhealthy. It is possible, however, to correct the corrupted link flows \textit{accurately} with the proposed method under a recoverability condition if there are only a few bad sensors which are located at certain links. We analytically identify the links that are robust to miscounts and relate them to the geometric structure of the traffic network by introducing the recoverability concept and an algorithm for computing it. The recoverability condition for corrupted links is simply the associated recoverability being greater than 1. In a more realistic setting, besides the unhealthy link sensors, small measurement noises may be present at the other sensors. Under the same recoverability condition, our method guarantees to give an estimated traffic flow fairly close to the ground-truth data and leads to a bound for the correction error. Both synthetic and real-world examples are provided to demonstrate the effectiveness of the proposed method.

Figures

Tables

Table 1. Table 1: Computational results for Example 1. Links with corrupted data are labeled with *.

Link ID	Ground-truth	Observation	Estimation	Estimation Error
1	10000	9950	9950	-10
2	70000	69887	69887	-113
3	8000	N/A	7953	-47
4	2000	1997	1997	-3
5	15000	15010	15104	104
6*	55000	39751	54783	-217
7	20000	20043	20043	43
8	3000	3014	3014	14
9	7000	6977	6977	23
10	9000	N/A	9009	9
11	5000	5045	5046	46
12	48000	47770	47771	-229
13	25500	25397	25505	5
14	1500	N/A	1515	15
15	20000	20000	20000	0
16*	33000	45302	32817	-183
17	45500	45912	45505	5
18	34500	34332	34332	-168

Table 2. Table 2: Computational results for Example 2.

Link ID	Observation	Estimation	Difference	Percentage Difference
1	123714	123714	0	0.0%
2	4835	4835	0	0.0%
3	N/A	128549	N/A	N/A
4	15479	15479	0	0.0%
5	105748	113070	7322	6.9%
6	11127	13661	2534	22.8%
7	127073	126731	-342	-0.3%
8	16194	16194	0	0.0%
9	110997	110537	-460	-0.4%
10	2809	2757	-52	-1.9%
11	113002	113295	293	0.3%
12	10941	10941	0	0.0%
13	N/A	124236	N/A	N/A
14	N/A	139715	N/A	N/A
15	124437	124322	-115	-0.1%
16	15393	15393	0	0.0%
17	113411	113413	2	0.0%
18	10907	10909	2	0.0%

Equations

$$A_{ij}=\begin{cases}-1&\mbox{if the $j$-th link is outgoing link of node i}\\ 1&\mbox{if the $j$-th link is incoming link of node i}\\ 0&\mbox{otherwise.}\\ \end{cases}$$

$$A\hat{f}=0_{(n)}.$$

$$f_{\mathcal{M}}={\hat{f}}_{\mathcal{M}}+e_{\mathcal{M}}\in {\mathbb{R}}^m,$$

$${\hat{f}}_{\mathcal{K}}=f_{\mathcal{K}}\text{mbox}and{\hat{f}}_{{\mathcal{K}}^c}=-(A^{{\mathcal{K}}^c}{)}^{-1}{A}^{\mathcal{K}}{f}_{\mathcal{K}}.$$

$$\hat{f}=Zx,\text{mbox}forsomex\in {\mathbb{R}}^{l-n}.$$

$$Z=\left[\begin{array}{c}{I}_{(l-n)}\\ -(A^{{\mathcal{K}}^c}{)}^{-1}{A}^{\mathcal{K}}\end{array}\right]\in {\mathbb{R}}^{l\times (l-n)}$$

$$x^{\ast }=\arg \underset{x\in {\mathbb{R}}^{l-n}}{\min }\Vert Z_{\mathcal{M}}x-f_{\mathcal{M}}{\Vert }_1,$$

$$f^{\ast }=Z{x}^{\ast }.$$

$$\mathrm{Rec}(\mathcal{S};A,\mathcal{M}):=\underset{h\in \mathrm{Ran}(Z):\Vert h_{\mathcal{S}}{\Vert }_1\ne 0}{\inf }\frac{\Vert h_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1}{\Vert h_{\mathcal{S}}{\Vert }_1},$$

$$\mathrm{Rec}(\mathcal{S};A,\mathcal{M})=\underset{v\in {\mathbb{R}}^{l-n}:Z_{\mathcal{S}}v\ne 0}{\inf }\frac{\Vert Z_{\mathcal{M}\setminus \mathcal{S}}v{\Vert }_1}{\Vert Z_{\mathcal{S}}v{\Vert }_1},$$

$$A=\left[\begin{array}{cccccc}1& 1& -1& -1& 0& 0\\ 0& 0& 1& 0& -1& 0\\ 0& 0& 0& 1& 1& -1\end{array}\right],$$

$$Z=\left[\begin{array}{ccc}-1& 0& 1\\ 1& 0& 0\\ 0& 1& 0\\ 0& -1& 1\\ 0& 1& 0\\ 0& 0& 1\end{array}\right].$$

$$Z_{\mathcal{M}}=\left[\begin{array}{ccc}-1& 0& 1\\ 1& 0& 0\\ 0& -1& 1\\ 0& 1& 0\\ 0& 0& 1\end{array}\right].$$

$$f^{\ast }=Z{x}^{\ast }=\left[\begin{array}{c}300\\ 200\\ 300\\ 200\\ 300\\ 500\end{array}\right]=\hat{f}.$$

$$\Vert f^{\ast }-\hat{f}{\Vert }_1\le \lambda (\alpha ,A,\mathcal{M})\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1,$$

$$\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1=2+1+2+1=6.$$

$$\Vert f^{\ast }-\hat{f}{\Vert }_1=12.$$

$${\left(\mathrm{shrink}(z,r)\right)}_i=\mathrm{sign}(z_i)\max \{|z_i|-r,0\},i=1,\dots ,m.$$

$$\underset{v\in {\mathbb{R}}^{l-n}}{\min }\frac{\Vert Z_{\mathcal{M}\setminus \mathcal{S}}v{\Vert }_1}{\Vert Z_{\mathcal{S}}v{\Vert }_1}.$$

$$f^{\ast }=\left[\begin{array}{c}{I}_{(l-n)}\\ -(A^{{\mathcal{K}}^c}{)}^{-1}{A}^{\mathcal{K}}\end{array}\right]x^{\ast }=\left[\begin{array}{c}{x}^{\ast }\\ -(A^{{\mathcal{K}}^c}{)}^{-1}{A}^{\mathcal{K}}{x}^{\ast }\end{array}\right].$$

$$v\in \mathrm{Ran}(Z),v_{\mathcal{M}}\in \mathrm{Ran}(Z_{\mathcal{M}})$$

$$\Vert f_{\mathcal{M}}^{\ast }-f_{\mathcal{M}}{\Vert }_1=\Vert Z_{\mathcal{M}}{x}^{\ast }-f_{\mathcal{M}}{\Vert }_1\le \Vert {\hat{f}}_{\mathcal{M}}-f_{\mathcal{M}}{\Vert }_1.$$

$$\Vert {\hat{f}}_{\mathcal{M}}-f_{\mathcal{M}}{\Vert }_1=\Vert e_{\mathcal{M}}{\Vert }_1=\Vert e_{\mathcal{S}}{\Vert }_1+\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1,$$

$$\Vert f_{\mathcal{M}}^{\ast }-f_{\mathcal{M}}{\Vert }_1$$

$$=\Vert v_{\mathcal{S}}-e_{\mathcal{S}}{\Vert }_1+\Vert v_{\mathcal{M}\setminus \mathcal{S}}-e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1$$

$$\ge \Vert e_{\mathcal{S}}{\Vert }_1-\Vert v_{\mathcal{S}}{\Vert }_1+\Vert v_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1-\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1$$

$$\Vert e_{\mathcal{S}}{\Vert }_1+\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1\ge \Vert e_{\mathcal{S}}{\Vert }_1-\Vert v_{\mathcal{S}}{\Vert }_1+\Vert v_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1-\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1$$

$$2\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1\ge -\Vert v_{\mathcal{S}}{\Vert }_1+\Vert v_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1$$

$$2\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1\ge \Vert v_{\mathcal{M}}{\Vert }_1-2\Vert v_{\mathcal{S}}{\Vert }_1\ge (1-\frac{2}{1+\alpha })\Vert v_{\mathcal{M}}{\Vert }_1,$$

$$\Vert (f^{\ast }-\hat{f}{)}_{\mathcal{M}}{\Vert }_1=\Vert v_{\mathcal{M}}{\Vert }_1\le \frac{2(\alpha +1)}{\alpha -1}\Vert e_{\mathcal{M}\setminus \mathcal{S}}{\Vert }_1.$$

$$0_{(n)}=A(f^{\ast }-\hat{f})=[A^{\mathcal{K}},A^{{\mathcal{K}}^c}]\left[\begin{array}{c}(f^{\ast }-\hat{f}{)}_{\mathcal{K}}\\ (f^{\ast }-\hat{f}{)}_{{\mathcal{K}}^c}\end{array}\right],$$

Full text

Keywords: Link flow correction; $\ell_{1}$-minimization; flow conservation law; recoverability; exact recovery; correction bound.

1 Introduction

Link volume/flow data is an important data source in both long-term planning and short-term operation applications. The examples include but are not limited to signal timing, toll road pricing, origin-destination trip matrix estimation, transportation planning, traffic safety (e.g. [14, 16, 19, 17] and the references therein).

The flow conservation in a traffic network implies that the total in-flow equals the total out-flow at each non-centroid node. The centroids are nodes where traffic originates/is destined to, and non-centroids nodes denotes all the other nodes. Practically, when looking at traffic flow counts over a sufficiently long time period (e.g. daily cumulative flow), we expect that the sum of cumulative link flows entering the non-centroid node equals the sum of cumulative link flows leaving it.

The flow conservation law is an important property, which has been exploited in many different applications. For example, the widely used first-order traffic flow model, the LWR model [15, 22], is derived based on the conservation of traffic. In [6], the authors mentioned that a path flow estimator (PFE) needs reasonably consistent link flows, meaning that the flow conservation law should be satisfied within a certain error bound, to reproduce feasible path flow solutions.

In practice, the flow conservation law can be violated due to numerous flow measuring errors; i.e., the observed flow counts are generally corrupted and cause data inconsistency issues. In [23], the network sensor health problem (NSHP) ([23]) is proposed to evaluate individual sensors’ health indices based on the level of flow data consistency. Assuming flow counting sensors are already installed on some of the links where at least one base set exists, the NSHP tries to find the least inconsistent base set that “minimizes the sum of squares of the differences between observed and calculated link flows”. The health index of a specific sensor is evaluated based on the frequency that it appears in the least inconsistent set.

Several studies have looked into the problem of correcting inconsistent flow data according to flow conservation. To solve a similar problem in transit planning, Kikuchi et al. [12] studied the passenger flow balancing problem and proposed a least square correction method to adjust the flows, so that the counts are conserved and close to the observed values. van Zuylen and Branston [25] assumed that the observed link flows follow probability distributions constrained by flow conservation. The study derived the formula for constrained maximum likelihood estimates of the link flows. Kikuchi et al. [13] examined and compared six different methods to adjust observed flow rate according to flow conservation. All of the methods have the same constraints but different objective functions. Vanajakshi and Rilett [24] studied flow inconsistency problem between neighboring upstream and downstream loop detectors. A nonlinear optimization problem is proposed to correct loop detector data, in the case when observed data violates flow conservation.

In summary, given the observed cumulative flows on different links, all of the existing flow correction methods adopted optimization approaches that try to meet the following principles:

• •

Ensure that flow conservation be followed exactly at all non-centroid nodes after adjustment using a set of constraints,

• •

Preserve the integrity of the observed data as much as possible by minimizing the distance between adjusted and observed flows.

However, all of the studies are limited to simple hypothetical networks or networks with simple topologies. Also, no systematic study has been done regarding the effectiveness and applicability of the methods.

In this study, we propose a method to estimate the true link flow from corrupted data on observed links as well as unobserved links via $\ell_{1}$-minimization. Similar to the existing methods, the link flow correction method is also formulated as an optimization problem to minimize the difference between observed and estimated link flows. As an improvement over the existing methods, the node-based formulation of flow conservation is introduced to handle general road network where link flows are only observed on monitored links, not on all links as assumed in many existing studies. More importantly, we adopt the $\ell_{1}$-minimization method from compressed sensing [4, 3] to analytically derive the condition for exact/stable recovery of the true cumulative flow counts.The $\ell_{1}$ norm is the unique convex sparsity promoting penalty. Though it is not differentiable, various efficient scalable numerical methods exist to date for its minimization [1, 2, 7, 9, 28] besides linear programming. In addition to $\ell_{1}$ norm, other non-convex sparsity promoting penalty functions can also be considered; see [29, 30, 18] and references therein. Their minimization is computationally more expensive than $\ell_{1}$, and we shall leave such a study for a future work.

The rest of the paper is organized as follows. In section 2, we state the link flow correction problem formulation, the exact and stable recovery theorem, the recoverability condition and the connection with compressed sensing. In section 3, we use a toy example to illustrate the conditions for exact and stable link flow recovery. In section 4, we use real-world loop detector data as an application for this method. In both the toy and real world examples, the recoverability condition is verified analytically. The concluding remarks are in section 5.

Notations

Let us fix some notations. $\mathbb{R}^{n}$ represents the real coordinate space of $n$ dimensions. Let $x\in\mathbb{R}^{n}$, $\|x\|_{1}:=\sum_{i=1}^{n}|x_{i}|$ takes the $\ell_{1}$ norm of $x$, and $\|x\|$ denotes the Euclidean ($\ell_{2}$) norm. Given any index set $\mathcal{I}\subseteq\{1,2,\dots,n\}$, $|\mathcal{I}|$ counts the number of elements in $\mathcal{I}$; $\mathcal{I}^{c}:=\{1,2,\dots,n\}\setminus\mathcal{I}$ is the complement set of $\mathcal{I}$. $x_{\mathcal{I}}\in\mathbb{R}^{|\mathcal{I}|}$ consists of the elements in $x$ restricted to the index set $\mathcal{I}$. $\mathbf{0}_{(n)}\in\mathbb{R}^{n}$ denotes the vector containing zeros only, while $I_{(n)}\in\mathbb{R}^{n\times n}$ denotes the identity matrix of order $n$. For any matrix $A\in\mathbb{R}^{m\times n}$, $A^{\top}$ is the transpose of $A$; $A_{\mathcal{I}}\in\mathbb{R}^{|\mathcal{I}|\times n}$ is the submatrix of $A$ restricted to the row index set $\mathcal{I}\subseteq\{1,2,\dots,m\}$, and $A^{\mathcal{I}}\in\mathbb{R}^{m\times|\mathcal{I}|}$ is the submatrix of $A$ restricted to the column index set $\mathcal{I}\subseteq\{1,2,\dots,n\}$; e.g., $A_{\{1,2\}}$ extracts the first two rows of $A$, and $A^{\{1,2\}}$ extracts the first two columns of $A$. ${\rm Ker}(A):=\{x\in\mathbb{R}^{n}:Ax=\mathbf{0}_{(m)}\}$ represents the kernel space of $A$, while ${\rm Ran}(A):=\{h\in\mathbb{R}^{m}:h=Ax\mbox{ for some }x\in\mathbb{R}^{n}\}$ represents the range space of $A$.

2 Methodology

2.1 Problem setup

Given a traffic network with non-centroid nodes only, the node-link incidence matrix $A\in\mathbb{R}^{n\times l}$ with $n$ being the number of nodes and $l$ the number of links, can be expressed as

[TABLE]

Then $A$ is always of full (row) rank as proved in [21], and traffic flow data $\hat{f}\in\mathbb{R}^{l}$ obeys the flow conservation:

[TABLE]

Suppose $\mathcal{M}\subseteq\{1,2,\dots,l\}$ is the set of links whose link flows are observed, and $|\mathcal{M}|=m$. We call $M$ as “monitored set” thereafter. We assume that

[TABLE]

is the observed inconsistent flow data corrupted by sensing errors $e_{\mathcal{M}}\in\mathbb{R}^{m}$.

The flow correction problem is to derive an estimate of $\hat{f}$, denoted by $f^{*}$, from the corrupted data $f_{\mathcal{M}}$. Here we impose an underlying assumption on $\mathcal{M}$ for the flow correction problem to be well-posed. We will need the concept of base set introduced in [23].

Assumption 2.1**.**

$\mathcal{M}$* contains at least one base set $\mathcal{K}\subseteq\mathcal{M}$, meaning that $|\mathcal{K}|=l-n$ and $A^{\mathcal{K}^{c}}\in\mathbb{R}^{n\times n}$ is invertible.*

For the consistent data $\hat{f}_{\mathcal{M}}=f_{\mathcal{M}}$ (i.e., $e_{\mathcal{M}}=\mathbf{0}_{(m)}$), of course we have $\hat{f}_{\mathcal{K}}=f_{\mathcal{K}}$ since $\mathcal{K}\subseteq\mathcal{M}$. Then the $\hat{f}$ can be uniquely recovered by performing [21, 23]:

[TABLE]

If $\mathcal{M}$ contains more than one base set, the $\hat{f}$ recovered in the above from different $f_{\mathcal{K}}$ will be consistent.

Assumption 2.1 is the sufficient and necessary condition for the whole link flows to be observable. It guarantees that the whole flow data can be deduced from at least one subset of the observed link flows. Without this assumption, however, some of the link flows cannot be estimated from available data and the problem is unsolvable [21], whether the measured flows are consistent or not.

2.2 Flow correction via $\ell_{1}$-minimization

Since $A$ is of full row rank, ${\rm Ker}(A)$ is an $(l-n)$-dimensional subspace of $\mathbb{R}^{l}$. Suppose $Z\in\mathbb{R}^{l\times(l-n)}$ is the matrix whose columns form a basis of ${\rm Ker}(A)$. Since $\hat{f}\in{\rm Ker}(A)$, we have

[TABLE]

As a result, $\hat{f}_{\mathcal{M}}$ must be of the form $Z_{\mathcal{M}}x$ for some $x\in\mathbb{R}^{l-n}$.

Remark 2.1**.**

Clearly the existence of $Z$ is non-unique, but $f^{*}$ is invariant to the choice of $Z$ and only depends on the structure of the traffic network. Indeed $f^{*}$ is the one in ${\rm Ran}(Z)$ whose restriction on $\mathcal{M}$ has the least absolute deviation from $f_{\mathcal{M}}$. So $f^{*}$ only depends on ${\rm Ran}(Z)$ which is same as ${\rm Ker}(A)$. Note that $A$ is the node-link matrix uniquely determined by the network structure.

The following result not only gives a concrete construction of $Z$, but also interprets $x^{*}$ in (2.2) as an estimate of $\hat{f}_{\mathcal{K}}$ for some base set $\mathcal{K}$ (not necessarily a subset of $\mathcal{M}$).

Theorem 2.1**.**

Let $\mathcal{K}$ be any base set. Without loss of generality, suppose $A$ is partitioned as $[A^{\mathcal{K}},\;A^{\mathcal{K}^{c}}]$ with $A^{\mathcal{K}^{c}}\in\mathbb{R}^{n\times n}$ being invertible. Then

[TABLE]

is a basis matrix of ${\rm Ker}(A)$. Moreover, by choosing such $Z$, $x^{*}$ from (2.2) is an estimate of $\hat{f}_{\mathcal{K}}$.

We will show the proof in Appendix C. Our proposed method consists of the following two steps:

We first solve an $\ell_{1}$-minimization problem:

[TABLE]

That is, we seek an estimate of $e_{\mathcal{M}}=f_{\mathcal{M}}-\hat{f}_{\mathcal{M}}$ in the affine space $\{f_{\mathcal{M}}-Z_{\mathcal{M}}x:x\in\mathbb{R}^{l-n}\}$ with the least $\ell_{1}$ norm. The problem (2.2) can be efficiently solved by the alternating direction method of multipliers (ADMM) [2]; see Appendix A for the implementation details.

• 2.

$\hat{f}$ is then estimated by

[TABLE]

$Zx^{*}$ may have non-integer entries, in this case, we can just perform rounding.

2.3 Connections with compressed sensing

Compressed sensing [3, 8] aims to recover a sparse signal (vector) $y$ from an under-determined linear system that generally has infinitely many solutions. It enables recovery of the signal $y$ from far fewer samples than required by the Nyquist-Shannon sampling theorem. Major ingredients of the standard compressed sensing technique include

• •

Sparsity: most of the entries in $y$ are zeros.

• •

$\ell_{1}$-minimization: minimizing $\|y\|_{1}$ to exploit the sparsity of $y$.

Let us return to the flow correction problem, which is in essence equivalent to the estimation of $e_{\mathcal{M}}$. In an extreme case, suppose all the sensors are bad, leading to large sensing errors. Without further information, it is clearly impossible to get a good estimate of $\hat{f}$ from $f_{\mathcal{M}}$ by any means. Intuitively, however, reconstructing $\hat{f}$ is promising if most of the sensors record consistent flow data. Mathematically speaking, $e_{\mathcal{M}}$ is sparse. The flow correction problem thus can be viewed as sparse error correction problem [4, 27], which is similar to compressed sensing. Note that, however, the flow correction problem deviates from the traditional compressed sensing problem, where the matrix $Z$ would be random.

3 Correction results

Note that our proposed method does not take advantage of any prior information about the possible bad sensors. Apparently one can not always hope for a good estimation $f^{*}$ to $\hat{f}$, even if there is only one bad sensor in the network. For instance, in the network shown in Figure 1, if the sensor on link 1 gives very wrong count, then basically there is no way to reasonably correct this error because links 1 and 2 are equivalent in the topology of the network. With that said, without extra information, obtaining a good estimate of $\hat{f}$ is possible only when the bad sensors are located at some particular links. These locations tolerating miscount are somehow determined by the network structure. In the following, we shall introduce the concept of recoverability.

Definition 3.1**.**

Given a network with node-link incidence matrix $A$ and monitored link set $\mathcal{M}$, we define the recoverability for the subset $\mathcal{S}\subseteq\mathcal{M}$ by

[TABLE]

which is a function of the subset $\mathcal{S}$ and also determined by both the network structure $A$ and the monitored link set $\mathcal{M}$.

Since $h=Zv$ for some $v\in\mathbb{R}^{l-n}$, then we can rewrite (3.4) as

[TABLE]

which resembles the classical Rayleigh quotient for the principal eigenvalue $\mu$ of the generalized eigenvalue problem [26]: $Z_{\mathcal{S}}^{\top}\,Z_{\mathcal{S}}\,v=\mu\,Z_{\mathcal{M}\setminus\mathcal{S}}^{\top}\,Z_{\mathcal{M}\setminus\mathcal{S}}\,v$ if $\ell_{2}$ norm replaces the $\ell_{1}$ norm. The optimization of the ratio of two homogeneous functions of degree one has been studied [10] where an inverse power iterative algorithm was proposed. Based on [10], we propose an efficient algorithm to solve problem (3.5) which will be detailed in Appendix B.

3.1 Exact recovery

We first consider the case where some sensors are bad, which introduce inconsistency of the flow data. The following Theorem 3.1 asserts that when the bad sensors are located at certain link set $\mathcal{S}$ whose size is expected to be small, then no matter how large the errors are, we are able to exactly recover $\hat{f}$ from $f_{\mathcal{M}}$.

Theorem 3.1** (Exact recovery).**

Let $\mathcal{S}:=\{i\in\mathcal{M}:e_{i}\neq 0\}$, which means miscounts only occur at the link set $\mathcal{S}$. If ${\rm Rec}(\mathcal{S};A,\mathcal{M})>1$, then the estimation $f^{*}$ computed by (2.3) is equal to $\hat{f}$. That is, the links in $\mathcal{S}$ are robust to miscounts if ${\rm Rec}(\mathcal{S};A,\mathcal{M})>1$.

The proof is omitted here, since the above theorem is a special case of Theorem 3.2 in section 3.2. We remark that the lower bound for ${\rm Rec}(\mathcal{S};A,\mathcal{M})$ in the recoverability condition is sharp. Indeed the correction method can fail when ${\rm Rec}(\mathcal{S};A,\mathcal{M})=1$, as will be seen in the following example.

Example 3.1**.**

Let us consider the traffic network associated with the 3$\times$6 node-link incidence matrix

[TABLE]

and the ground-truth network flow $\hat{f}=\begin{bmatrix}300\\ 200\\ 300\\ 200\\ 300\\ 500\end{bmatrix}$ as in Figure 1, the node and links are labeled with their ID with ground truth link flows in the parentheses.

Then Theorem 2.1 gives that

[TABLE]

Let the monitored link set be $\mathcal{M}=\{1,2,4,5,6\}$, then

[TABLE]

Let the observation be $f_{\mathcal{M}}=\begin{bmatrix}f_{1}\\ f_{2}\\ f_{4}\\ f_{5}\\ f_{6}\end{bmatrix}=\begin{bmatrix}300\\ 200\\ 200\\ 300\\ 600\end{bmatrix}$, i.e., the observed link flow on link 6 is inflated by 100 due to sensor error. So $e_{\mathcal{M}}=f_{\mathcal{M}}-\hat{f}_{\mathcal{M}}=\begin{bmatrix}0\\ 0\\ 0\\ 0\\ 100\end{bmatrix}$, $\mathcal{S}=\{i\in\mathcal{M}:e_{i}\neq 0\}=\{6\}$, and $\mathcal{M}\setminus\mathcal{S}=\{1,2,4,5\}$.

We can verify by either an analytic approach or Algorithm 2 that the recoverability condition ${\rm Rec}(\mathcal{S};A,\mathcal{M})=2>1$ is satisfied. Then Theorem 3.1 asserts that $f^{*}$ derived from (2.2) and (2.3) must be equal to $\hat{f}$. It is indeed true because $x^{*}=\begin{bmatrix}200\\ 300\\ 500\end{bmatrix}$, and therefore

[TABLE]

Compare this result with the ground truth link flows, we can conclude that the errors are completely eliminated.

Remark 3.1**.**

We have two remarks below.

• •

Without knowing the count at link 3, i.e., $\mathcal{M}=\{1,2,4,5,6\}$, the proposed method would fail exact recovery if the count was corrupted at any other link except link 6. Take link 1 for example, it is easy to check that $\mathrm{Rec}(\{1\};A,\mathcal{M})=1$. Therefore, link 1 is not guaranteed to be robust to miscount by our theory. Indeed this is the case as mentioned in the beginning of this section.

• •

Suppose link 3 was also monitored, i.e., $\mathcal{M}=\{1,2,\dots,6\}$, then any counting error at one of the links 3, 4, 5 and 6 could be accurately corrected by our method.

3.2 Stable recovery

In a more realistic setting, we assume that all the elements in $e_{\mathcal{M}}$ are non-zeros, yet most of them are relatively small compared with the other few. This refers to approximate sparsity in compressed sensing. In this case, it is still possible for $f^{*}$ to be close enough to $\hat{f}$. In another word, the estimation errors are bounded from above in this case.

Theorem 3.2** (Stability).**

For any $\mathcal{S}\subseteq\mathcal{M}$, if ${\rm Rec}(\mathcal{S};A,\mathcal{M})=\alpha>1$, then $f^{*}$ computed by (2.3) obeys

[TABLE]

for some constant $\lambda(\alpha,A,\mathcal{M})>0$ depending only on $\alpha$, $A$ and $\mathcal{M}$. Moreover, $\lambda(\alpha,A,\mathcal{M})$ decreases in $\alpha$, meaning that larger recoverability leads to higher correction accuracy.

In view of (3.6), $f^{*}$ is a good estimation if $\|e_{\mathcal{M}\setminus\mathcal{S}}\|_{1}$ is small. On the other hand, the estimation error does not rely on $e_{\mathcal{S}}$. Theorem 3.1 is essentially a corollary of Theorem 3.2 in the special case $\|e_{\mathcal{M}\setminus\mathcal{S}}\|_{1}=0$. Therefore, it suffices to prove Theorem 3.2 only. The proof of Theorem 3.2 will be detailed in Appendix C, in which we derive an explicit expression for $\lambda(\alpha,A,\mathcal{M})$.

Example 3.2**.**

We consider the same setting as in Example 3.1 except that the other observed data contains small sensing noise besides the large corruption at link 6. Specifically, let $f_{\mathcal{M}}=\begin{bmatrix}302\\ 201\\ 198\\ 301\\ 600\end{bmatrix}$ and $e_{\mathcal{M}}=f_{\mathcal{M}}-\hat{f}_{\mathcal{M}}=\begin{bmatrix}302\\ 201\\ 198\\ 301\\ 600\end{bmatrix}-\begin{bmatrix}300\\ 200\\ 200\\ 300\\ 500\end{bmatrix}=\begin{bmatrix}2\\ 1\\ -2\\ 1\\ 100\end{bmatrix}$. Again we take $\mathcal{S}=\{6\}$. Since $\mathrm{Rec}(\mathcal{S})=2>1$, it is asserted by Theorem 3.2 that the $\ell_{1}$ norm of the estimation error $\|f^{*}-\hat{f}\|_{1}$ is comparable to

[TABLE]

This is true, because $x^{*}=\begin{bmatrix}201\\ 303\\ 503\end{bmatrix}$ by (2.2), $f^{*}=\begin{bmatrix}302\\ 201\\ 303\\ 200\\ 303\\ 503\end{bmatrix}$ , and

[TABLE]

Note that the original counting error at Link 6 is 100, in sharp contrast to the error by our correction method which is just 3.

4 Test Examples

In this section, we provide both synthetic and real-world examples to demonstrate effectiveness of our proposed method.

4.1 A synthetic network

Figure 2 shows a parallel highway network [11, 21] with 9 nodes and 18 links among which 15 links are monitored. We create the ground-truth, observed and estimated flow data and list them in Table 1. The data on links 3, 10 and 14 are unobservable. They are marked by “N/A” in the table and by dashed line in the plot. The recorded data on links 6 and 16 are severely corrupted, while the other data contain small noise. So basically $\mathcal{M}=\{1,2,4,5,6,7,8,9,11,12,13,15,16,17,18\}$ and $\mathcal{S}=\{6,16\}$. It is clear that our estimation by Algorithm 1 is fairly close to the ground-truth, and the miscounts on links 6 and 16 are successfully detected. In fact, we can check by Algorithm 2 that the recoverability condition $\mathrm{Rec}(\mathcal{S};A,\mathcal{M})=1.5>1$ holds. Therefore, Theorem 3.2 provides guarantee for our correction result.

4.2 A real-world example

The daily cumulative flow data in this example is from Caltrans Performance Measurement System (PeMS) database, collected on I-405 northbound in the city of Irvine, on April 28, 2016. The network has 18 links and 9 nodes as illustrated in Figure 3. The loop detectors are installed on all links except for links 3, 13, and 14, which are represented by dashed lines. The links are labeled with their IDs and corresponding observed flows in the parentheses.

The estimated link flows by (2.2) and (2.3) are compared with the observed link flows in Table 2, where unobserved links flows are marked by “N/A”. Our correction result shows that the estimation error at link 6 is much larger than all other links. Since there is no ground-truth data available in this example, we can not check the correction quality directly. However, link 6 is flagged as unhealthy sensor by PeMS, which is consistent with our estimation. On the other hand, if link 6 is indeed the only unhealthy sensor, the quality of the estimated link flow listed in Table 2 is guaranteed by Theorem 3.2, in which we have $\mathcal{M}=\{1,2,4,5,6,7,8,9,10,11,12,15,16,17,18\}$ and $\mathcal{S}=\{6\}$. It can be verified that the recoverability condition ${\rm Rec}(\mathcal{S};A,\mathcal{M})=2>1$ holds.

5 Conclusion

In this study, we systematically studied the link flow correction problem in a traffic network based on flow conservation. The problem is formulated as an $\ell_{1}$-minimization problem, in which the differences between the estimated and observed link flows are minimized. We introduced the recoverability concept for a subset of links and specifically derived the recoverability condition for exactly retrieving the missing data: when certain sensors are malfunctioning, no matter how large the errors are, the ground truth flow can be exactly recovered. That is, some links are robust to miscounts. Furthermore, when small errors are present in observed link flows, the estimation error bound is found such that we can estimate the link flows that are close enough to ground-truth under the recoverability condition. We also showed an efficient algorithm for computing recoverability.

A few follow-up study topics can be interesting both theoretically and practically. In addition to the $\ell_{1}$ norm, it will be interesting to investigate the feasibility and efficiency of other sparsity promoting penalty functions for formulating and solving the flow correction problem. The recoverability defined in (3.4) is central to the flow correction problem, as it determines whether exact recovery is possible or not (see Theorem 3.1) and also the error bound in stable recovery (see (3.6)). In the future we will be interested in examining with Algorithm 2 how the road network’s structure impacts the recoverability of a subset of links, and such a study could provide guidelines for installing flow counting sensors especially in a large-scale network.

Acknowledgments

Yin and Xin were partially supported by NSF grants DMS-1522383 and IIS-1632935. Yin was also supported by ONR grant N000141617157. We would like to thank the referees for their constructive comments.

Appendix A. ADMM for solving (2.2)

The following alternating direction method of multipliers (ADMM) [2] is an thresholding-based iterative algorithm. In Algorithm 1, $z\in\mathbb{R}^{m}$ and $u\in\mathbb{R}^{m}$ are auxiliary variables. ’shrink’ is the so-called soft-thresholding operator on $\mathbb{R}^{m}$. For any $z\in\mathbb{R}^{m}$ and $r>0$, $\mathrm{shrink}(z,r)$ performs component-wise operation on $z$ given by

[TABLE]

The algorithm stops after some maximum number of iterations.

Appendix B. An inverse power algorithm for solving (3.5)

We present Algorithm 2 to solve the following optimization problem (3.5):

[TABLE]

The output $\lambda^{*}$ is the optimal objective value in (3.5), i.e., $\mathrm{Rec}(\mathcal{S};A,\mathcal{M})$. Note that in Algorithm 2, updating $v$ under the unit ball constraint is non-trivial and requires extra effort. We write an ADMM solver for this subproblem in Algorithm 3 below.

Appendix C. Technical proofs

Proof of Theorem 2.1.

To prove $Z=\begin{bmatrix}I_{(l-n)}\\ -(A^{\mathcal{K}^{c}})^{-1}A^{\mathcal{K}}\end{bmatrix}\in\mathbb{R}^{l\times(l-n)}$ gives a basis of ${\rm Ker}(A)$, it suffices to show that

$AZ=O$ is a zero matrix. It is true since $AZ=[A^{\mathcal{K}},\;A^{\mathcal{K}^{c}}]\begin{bmatrix}I_{(l-n)}\\ -(A^{\mathcal{K}^{c}})^{-1}A^{\mathcal{K}}\end{bmatrix}=A^{\mathcal{K}}-A^{\mathcal{K}^{c}}(A^{\mathcal{K}^{c}})^{-1}A^{\mathcal{K}}=O$.

• 2.

$Z$ has full rank, i.e., $\mathrm{rank}(Z)=l-n$. This is also true because, on one hand $\mathrm{rank}(Z)\leq l-n$, on the other hand, $\mathrm{rank}(Z)\geq\mathrm{rank}(I_{(l-n)})=l-n$ since $I_{(l-n)}$ is a submatrix of $Z$.

Then by (2.3), we have

[TABLE]

Since $\hat{f}=\begin{bmatrix}\hat{f}^{\mathcal{K}}\\ \hat{f}^{\mathcal{K}^{c}}\end{bmatrix}$, we conclude that $x^{*}$ is an estimate of $\hat{f}^{\mathcal{K}}$. ∎

Proof of Theorem 3.2.

Suppose $f^{*}=\hat{f}+v$, since $f^{*},\;\hat{f}\in{\rm Ran}(Z)$, then we have

[TABLE]

Moreover, since $f_{\mathcal{M}}^{*}=Z_{\mathcal{M}}x^{*}$ and $\hat{f}_{\mathcal{M}}\in{\rm Ran}(Z_{\mathcal{M}})$, (2.2) implies that

[TABLE]

Keep in mind that $e_{\mathcal{M}}=f_{\mathcal{M}}-\hat{f}_{\mathcal{M}}$ is the sensing error, so on the right hand side of (5.7),

[TABLE]

and on the left hand side,

[TABLE]

In (5.9), we used the triangle inequality for $\ell_{1}$ norm. Combining (5.7), (5.8), and (5.9), we have

[TABLE]

or

[TABLE]

By the assumption that ${\rm Rec}(\mathcal{S};A,\mathcal{M})=\alpha>1$, we have $\alpha\|h_{\mathcal{S}}\|_{1}\leq\|h_{\mathcal{M}\setminus\mathcal{S}}\|_{1}$ holds for all $h\in{\rm Ran}(Z_{\mathcal{M}})$. Since $v_{\mathcal{M}}\in{\rm Ran}(Z_{\mathcal{M}})$ as aforementioned, we have $\|v_{\mathcal{M}}\|_{1}\geq(1+\alpha)\|v_{\mathcal{S}}\|_{1}$, then it follows from (5.10) that

[TABLE]

and thus

[TABLE]

In what follows, we derive an upper bound for $\|(f^{*}-\hat{f})_{\mathcal{M}^{c}}\|_{1}$. Without loss generality, suppose $A=[A^{\mathcal{K}},\;A^{\mathcal{K}^{c}}]$ with $\mathcal{K}\subseteq\mathcal{M}$ being any base set. Since both $f^{*}$ and $\hat{f}$ obey flow conservation, we have

[TABLE]

which gives

[TABLE]

Since $\mathcal{K}\subseteq\mathcal{M}$, we have $\mathcal{M}^{c}\subseteq\mathcal{K}^{c}$. Therefore, $(f^{*}-\hat{f})_{\mathcal{K}}$ is contained in $(f^{*}-\hat{f})_{\mathcal{M}}$, and $(f^{*}-\hat{f})_{\mathcal{M}^{c}}$ is contained in $(f^{*}-\hat{f})_{\mathcal{K}^{c}}$. Using the above facts, we have

[TABLE]

In the second inequality above, $\|(A^{\mathcal{K}^{c}})^{-1}A^{\mathcal{K}}\|_{1}$ is the operator norm of $(A^{\mathcal{K}^{c}})^{-1}A^{\mathcal{K}}$ induced by $\ell_{1}$ norm. And in the last inequality, we used (5.11).

Finally, combining (5.11) and (Proof of Theorem 3.2.) gives that

[TABLE]

Note that the above inequality holds for all base set $\mathcal{K}\subseteq\mathcal{M}$. Therefore,

[TABLE]

which concludes the proof. ∎