Faster and Smaller Two-Level Index for Network-based Trajectories

TL;DR

This paper introduces a compact data structure for the bottom level of two-level network trajectory indexes, significantly improving speed and size efficiency over existing methods.

Contribution

It presents a novel, specialized data structure for the interval-intersection problem in network trajectory indexing, outperforming non-specialized spatial indexes.

Findings

Faster query processing times

Reduced index size

Improved overall efficiency

Abstract

Two-level indexes have been widely used to handle trajectories of moving objects that are constrained to a network. The top-level of these indexes handles the spatial dimension, whereas the bottom level handles the temporal dimension. The latter turns out to be an instance of the interval-intersection problem, but it has been tackled by non-specialized spatial indexes. In this work, we propose the use of a compact data structure on the bottom level of these indexes. Our experimental evaluation shows that our approach is both faster and smaller than existing solutions.

Figures

Full text

1 Introduction

Spatio-temporal information has gained popularity in decision making systems, such as optimization of transportation systems, urban planning, and so on. The proliferation of different types of sensors to capture or generate this kind of data has made these applications possible but, at the same time, it has also made challenging the storage and processing of spatio-temporal data. The work in this paper focuses on a subcategory of spatio-temporal data, that is, trajectory of moving objects, which can be be reconstructed by the GPS devices of smart-phones or, at a different granularity, by smart transportation cards.

Trajectories can be classified as free-trajectories, in which movement is not constrained, and network-based trajectories, in which movement is constrained to a network and cannot exist outside such network. Hurricanes and animal migrations are examples of the former, whereas public transportation is an example of the latter. Useful queries that can be answered by handling trajectories are: count the number of vessels inside a region during a time period (e.g. fishing closed season) or find the shortest path between two stops of a transportation system during a time period.

Several spatio-temporal indexes have been proposed to handle both free and network-based trajectories. However, classical solutions to deal with moving-object data are inefficient when facing the data volume collected through new sensor technology and the increasing interest for data analysis. On the other hand, space-efficient data structures have been proved to be successful for handling large volumes of data in many different domains, such as the Web, biological sequences, documents and code repositories, to name some examples.

In this work, we focus on two-level indexes for network-based trajectories and propose a new solution that uses compact data structures on the bottom level. This approach turns out to be smaller and faster than existing solutions.

2 Background and Related Work

A data structure for trajectories must provide access methods that allow the processing of spatio-temporal queries. These queries can be classified into coordinate- and trajectory-based queries [25]. Coordinate-based queries include time-slice queries that determine the position of objects at a given time instant, time-interval queries that extend time-slice queries to a time range, and queries about nearby neighbors. As for trajectory-based queries, they include topological queries, which involve information regarding the movement of an object, and queries related to navigation, which involve information derived from the movement, such as speed or direction. There also exist combinations such as “Where was object $X$ at a given time instant”.

Various data structures have been proposed to efficiently support queries on trajectories. These structures can be broadly classified into two categories: i) Data structures to support free movements on a space, such as 3D R-tree (a three-dimensional extension of the R-tree [17]), TB-tree [25] (which preserves the trajectories while allowing typical range queries on an R-tree) and MV3R-tree [31] (which uses a multi-version R-tree, called MVR-tree, along with an auxiliary 3D R-tree). ii) Data structures to support movements on networks, such as FNR-tree [14] (which uses a combination of a 2D R-tree with a forest of 1D R-trees), MON-tree [1] (using 2 levels of 2D R-trees) and PARINET [27] (based on graph partitioning and the use of B+-tree). Among the previous structures, FNR-tree and MON-tree have in common the separation of spatial and temporal dimensions, using a spatial structure (two-dimensional) and a forest of temporal structures (one-dimensional) to tackle each of these sub-problems separately.

Like FNR-tree and MON-tree, we focus on these two-level indexes. To solve the spatial problem, that is, the representation of the network in space (two-dimensional plane), aforementioned structures use a 2D R-tree, storing the segments of the network as lines. With the spatial problem solved, time has to be associated with segments in the network. More precisely, it is necessary to look for all the time intervals (times in which some objects pass through a segment) that intersect with a given query interval. This problem is known in the literature as interval intersection, an extension of the interval stabbing problem [29]. Classical structures to solve this problem are Interval trees and Priority trees [5].

As for the subproblem in the temporal dimension, FNR-tree makes use of a one-dimensional R-tree for each segment. These 1D R-trees index the objects whose trajectories pass through the segments of the network, storing the instant they enter and leave the segment in the form of a time interval $(t_{entry},t_{exit})$. Since only these intervals are stored, the structure assumes that objects do not stop or change speed or direction in the middle of a segment, they can only do so at nodes. MON-tree eliminates this restriction by replacing the one-dimensional R-trees with two-dimensional R-trees, where they store the relative movement within the segments as rectangles in the 2D R-tree of the form $(p_{1},p_{2},t_{1},t_{2})$, with $(p_{1},p_{2})$ a range of relative positions and $(t_{1},t_{2})$ a temporal interval.

While some of aforementioned structures support queries efficiently on large datasets, they are incapable of handling the increasing data volume of current applications. This has forced the use of compression techniques for data storage and transmission. Some techniques are to reduce the number of points in a curve [22] or to use features at each point, such as speed and orientation [26]. Both techniques work in free spaces and, when the movement is restricted to networks, it is even possible to get a better compression, like the ones shown in [18, 19, 20, 28].

Previous compression techniques improve storage requirements and transmission time of large datasets. However, the compression can be directly exploited by data structures that can maintain a compact representation of the data while allowing for indexed search capabilities. These structures have been called self-indexes and have been successfully implemented in other domains, such as information retrieval [23].

Recently, compact data structures have been also used for the representation of trajectories. GraCT [9], for free paths, uses a k2-tree [10] to store the absolute position of the objects in regular time intervals (snapshots) plus compressed logs for the representation of the movements between snapshots. ContaCT [8] improves GraCT with more efficient logs. Both structures answer spatio-temporal queries where space and time are the main filters, such as, “finding trajectories that went through a specific region at a given time instant”. On the other hand, CTR [7] supports trajectories restricted to networks by combining compressed suffixes arrays (CSA), to represent the nodes on the network an object passes through, and a balanced Wavelet matrix for the temporal component of the movement. In CTR, trajectories (or trips) are defined as sequences of labels, which represent the nodes of the network. Hence, it solves other types of queries in which the space is represented with such labels, such as “find the number of trajectories that started at $X$ and ended at $Y$”. This is a fundamental difference with our proposal, in which the spatial dimension are coordinates in a two-dimensional space, and not labels. This is also the main difference with CiNCT [20], which boosts CTR in terms of memory storage and query time.

Another difference with previous solutions is that our approach uncouples the network from the trajectories. This model known as Network-Matched has been successfully used [12, 21], but without using compact data structures in its implementation. Our approach has the advantage that mapping trajectories to a network facilitates the finding of similar trajectories and, in consequence, it allows a better use of space.

3 Data Structures for Network-based Trajectories

Similarly to the FNR-tree and MON-tree, we propose an index with two levels: spatial (top level) and temporal (bottom level). In a preliminary experimental evaluation, we observed that the spatial level requires negligible space compared with the temporal level. For example, for the Oldenburg network (see Section 4), in the baseline structure, the temporal level uses about 89% of the total memory with 1,000 objects circulating and about 94% with 2,000 objects. The more data are stored, the more negligible the spatial level becomes (due to the almost-static nature of the transportation network in comparison with the moving objects). Hence, we focus on optimizing the temporal level and process the spatial level with a two-dimensional R-tree, as the FNR-tree and MON-tree do. Recall that the R-tree is a balanced tree in which each leaf stores an entry of the form (id, MBB), where id is a reference to the data (in this case to a temporal index) and MBB is the Minimum Bounding Box that covers the spatial object (a line segment, in this domain). The R-tree does not provide worst-case guarantees as it may be forced to examine the entire tree in $O(n)$ time, even when the output is empty. However, it performs well in practice and is ubiquitous in spatial databases.

Each leaf of the R-tree contains a reference to a temporal index. These indexes solve the Interval Intersection problem. Before presenting alternatives to solve this problem, we give an overview of a query algorithm for spatio-temporal range queries, which are the most general coordinate-bases queries. First, a spatial query, a 2D window, is solved on the 2D R-tree, which returns a set of leaves whose segment may intersect the window. As in most spatial indexes, a refinement step is then executed to eliminate false positives, i.e. network segments whose MBBs intersect the window, but they do not actually intersect the window. After this refinement, the interval intersection query is executed in each temporal index referenced by the remaining leaves of the R-tree. Results from all these temporal indexes are then combined using an implementation of a set.

3.1 Temporal Level: Data structures for the Interval Intersection Problem

Unlike the FNR-tree and MON-tree, which use variants of an R-tree, we explore the use of specialized data structures for the interval-intersection problem.

3.1.1 Interval-tree [5].

This is a binary tree that is constructed recursively in the following way: i) The median $x_{med}$ of all the interval endpoints is computed. ii) Intervals are classified in three sets, $I_{med}$, $\mathit{I_{left}}$ and $I_{right}$, which contain intervals stabbed by $x_{med}$, intervals to the left of $x_{med}$ and intervals to the right of $x_{med}$, respectively. iii) $I_{med}$ is stored in a structure composed of two arrays sorted by left and right endpoints, and associated with the root, whereas $\mathit{I_{left}}$ and $I_{right}$ are recursively processed and assigned as left and right child, respectively.

A search for the intervals that intersect with the query interval $(l_{q},r_{q})$ is solved recursively starting from the root. The intervals within the visiting node that intersect the query interval are returned and the search is continued in the left child if $l_{q}$ is less than $x_{med}$ and/or in the right child if $r_{q}$ is greater than $x_{med}$. This data structure requires linear space and $O(\log n+k)$ query time, where $k$ is the number of reported results.

3.1.2 Schmidt.

The structure presented in [29] to solve the Interval Stabbing problem can be extended to solve also the Interval Intersection problem [11]. It defines the father of an interval as the rightmost interval among those that cover it completely. This relation forms a tree where siblings are ordered from left to right, and the root of the tree is a special node that acts as the father of all the intervals that are not covered by any other. In addition, for each possible endpoint of an interval, the structure stores an array called start, with a pointer to the node representing the rightmost-starting interval that intersects such point, and an array start2, storing a pointer to the node representing the rightmost interval starting up to such point (which may not be stabbed by it).

To solve an interval intersection query $q$, the algorithm first reports the rightmost interval that intersects $q$, which is $\max(start[l_{q}],start_{2}[r_{q}])$, if it exists. Then, the algorithm recursively reports the siblings to the left of the node while its right endpoint is greater than or equal to $l_{q}$, also searching among the right children of the reported nodes. This structure requires linear space and optimal $O(1+k)$ query time. Note, however, that this solution works only for small integer ranges. In order to work with intervals whose endpoints are floats, these endpoints are stored in sorted arrays and two binary searches are used to translate the query to rank space [11], which results in a total complexity of $O(\log n+k)$.

3.1.3 Compact data structure based on Independent Interval Sets (IIS).

A set of intervals $I=\{i_{1},i_{2},...,i_{n}\}$ is called an Independent Interval Set if no interval $i_{j}\in I$ is contained in any other interval $i_{k}\in I$.

Report the $k$ intervals of an IIS that intersect a query interval $Q=[l_{q},r_{q}]$ can be easily computed if we have the intervals in order. Note that, by definition of IIS, the order of the left endpoints of the intervals is the same as that of the right endpoints. If the first and the last interval intersected by the query are located, it is enough to iterate between them to return all the intersected intervals (see Figure 1).

In order to locate these two intervals, we could store the left and right coordinates of the intervals in two sorted arrays and use binary search to locate them, which is similar to what we did in previous solution. However, for this domain, we propose a simple solution that facilitates the use of compact data structures. Recall that the endpoints of our intervals are timestamps represented as float numbers. We multiply these timestamps by a scale factor to convert them to integers. For example, if we work with timestamps with up to $6$ decimals it is enough to multiply each one of them by $10^{6}$ to discretize the space. With this procedure, we obtain integer endpoints in an universe $U$ and, the larger the scale factor, the larger the universe.

After this discretization, we use two bitvectors, one for the left endpoints, start, and another for the right endpoints, end, of each interval in the set (see Figure 1). A $1$-bit in these bitvectors indicate that an interval starts (or ends, respectively), at such position. Then, for a query $Q=[l_{q},r_{q}]$ also discretized to this universe, two rank operations on these bitmaps are used to locate the first and last intervals intersected by the query: $rank_{1}(end,l_{q})$ and $rank_{1}(start,r_{q})$, respectively. As we mentioned above, the larger the scale factor, the larger the size of the universe $u$, which is the number of bits in these bitmaps. However, the number of set bits in them is $n$, which is the number of intervals (independently of the scale factor). Hence, we use the Elias-Fano representation [24] for this bitmaps, which takes $2n+n\log\frac{u}{n}$ bits of space. Note that, for a constant $c$ and $u=O(n^{c})$, it uses linear space as previous structures. The query time of rank operations on these bitmaps is $O(\log\frac{u}{n})$, thus, this structure can report the $k$ intervals intersecting the query in $O(\log\frac{u}{n}+k)$ time.

Although this solution only works for IIS, a general set of intervals can be decomposed into $m$ independent sets in $O(n\log m)$ time, for example, with Fredman’s algorithm [13] to find the optimal number of shuffled upsequences in a permutation (by considering the rightmost endpoints of the intervals as the permuted values). This leads to a solution that requires $O(m\log\frac{u}{n}+k)$ time to report the $k$ solutions. This does not provide worst case guarantees as $m$ can be as large as $n$, however, this adaptive analysis shows that this is an efficient solution for domains in which $m$ is small. The empirical evaluation in next section shows that this is precisely the case in our domain.

4 Experimental Results

All the implementations evaluated in this paper were coded in C++11. For the baselines, we use some available implementations: R-tree [2], Interval-tree111This implementation uses sequential search in each node, which is not optimal in theory, but performs well in practice. [15] and Schmidt [30]. We also make use of some succinct data structures from the SDSL library [16]. The experiments were run in a computer with an Intel Xeon E3-1220 v5 of 3.00 GHz CPU, 64GB of RAM, and implementations were compiled with g++ 5.4.0 over Ubuntu 16.04 (64 bits).

We first evaluate the performance of all the implementations for interval intersection on synthetic datasets, and then, the best candidates are evaluated in the complete solution for network-based trajectories.

4.1 Evaluation of Interval Intersection Data Structures

We evaluated the performance in three scenarios with different types of intervals: i) fixed size (Figure 2), ii) random size (Figure 3), and iii) intervals of trajectories extracted from a trajectories dataset generated with Brinkhoff’s generator [6] over San Francisco’s network (Figure 4). For each of these scenarios, we created a dataset with 800,000 intervals and a queryset with 500 random queries. Reported query time is the total time to solve all the queries.

Figure 2 shows the performance of the structures using fixed size intervals. The compact data structure shows the best performance among the four structures, with a considerable advantage in both query time and memory usage. In this scenario, intervals do not fully cover each other (except for precision issues), which produces a low number of independent sets in the IIS structure (only 6 for 800,000 intervals). This explains the outstanding performance of IIS.

Figure 3 shows the performance of the structures for random size intervals. The compact data structure keeps the best results in query time and memory usage (although in a tie with the Interval-tree) while the building time is drastically increased (up to 900 times the building time of the Interval-tree). This is explained by the high number of independent sets (3,273 for 800,000 intervals), which is caused by the frequency with which intervals fully cover each other.

Figure 4 shows the performance of the structures using time intervals extracted from synthetic trajectories obtained with Brinkhoff’s generator. The compact data structure shows a performance in between the two previous cases, but more similar to the first one. This shows the sensibility of the structure to the number of independent sets. In this dataset, intervals of trajectories have often similar length, producing a relatively low number of independent sets (29 for 800,000 intervals). Each temporal index is associated with a segment of the network, and moving objects usually traverse a same segment at a similar speed.

We also evaluated the sensibility of the structures to the scale used to transform original float-number times to integers. In this procedure, each time is multiplied by a scale factor and then truncated. In our datasets, original times use up to 8 digits to the right of the decimal point. Hence, a scale factor of $10^{8}$ guarantees a lossless transformation, whereas lower scale factors may produce a lossy transformation. In these experiments, we used the same 800,000 intervals of trajectories of the previous evaluation and results are shown in Figure 5.

Query time shows an almost constant behavior, except for the increase suffered by Schmidt, which is caused by the high number of duplicates when only 2 or less digits are used for the fractional part. In terms of space and construction time, the compact data structure is more sensible than the other structures, which is caused by the scale process. As we explain in previous section, the larger the scale factor, the larger the size of the bitmaps in this structure. Even so, this structure obtains the best results in both query time and memory usage, also giving the possibility to improve the performance in applications where the user can afford losing some precision. Note, however, that in all the other experiments we consider all the decimals, which is the worst case for our proposal.

4.2 Overall evaluation

From the experiments in previous section, we conclude that Schmidt’s structure is always outperformed by the others, and thus it is not considered in the implementation of data structures for trajectories. In the following experiments we compare our proposal, based on compact data structures, with two baselines: the original FNR-tree and an ad-hoc baseline in which 1D R-trees are replaced by interval trees. Note that in these experiments we are comparing three two-level indexes, all of them using a 2D R-tree on the top level.

The datasets of trajectories were created using Brinkhoff’s generator [6] over the real road networks of Oldenburg and San Francisco. The former consists of 6,105 nodes and 7,305 edges, whereas the latter consists of 175,343 nodes and 223,343 edges. We created trajectories for 1,000, 2,000, 3,000, 4,000 and 5,000 objects during 100 units of time for both networks.

4.2.1 Memory usage.

Figure 6 shows the space required by each of the structures. The proposed space-efficient solution (labeled as IIS in the graphs) obtained the best results in all the experiments. In addition, the larger the number of objects moving over the network, the larger the advantage of this structure over the baselines. For small number of moving objects, the total space used by the data structures is dominated by the spatial level, however, as this number increases, the temporal level dominates, and our proposal takes more advantage.

[math]1$$2$$3$$4$$5[math]5$$10$$15$$20$$25Num. of objects ($\times 10^{3}$)Memory usage (MB)OldenburgFNR-treebaselineIIS

[math]1$$2$$3$$4$$5[math]50$$100$$150Num. of objects ($\times 10^{3}$)San FranciscoFNR-treebaselineIIS

[TABLE]

The approximated memory usage per object is shown in Table 1, which shows that our approach requires about $70\%$ less memory than the FNR-tree, and about $60\%$ less memory than the baseline, when there are 5,000 objects moving over the networks. The difference between the two datasets is explained by the size of the network, the San Francisco network being much larger. First, part of the space charged to each object is due to the spatial index. However, the size of the network has also an impact on the distribution of objects per edge of the network. As this distribution is very skewed, the larger the network, the larger the number of nodes with few objects, which means an overhead.

4.2.2 Query Time.

The time performance of the structures was evaluated for three types of queries, which are the same used in the original evaluation of the FNR-tree [14]: i) Range Queries with Equal Spatial and Temporal Extent, such as “find all objects within a given area during a given time interval”; ii) Range Queries with Larger Temporal Extent, which query for very large time intervals, including intervals expanding the whole temporal dimension, such as “find all the objects having ever passed through a given area”; and iii) Time Slice Queries, that only consider a time instant, such as “find all the objects that were in a given area at a given time instant”. For each of these scenarios, we created three query-sets with 500 random queries for each network.

Figure 7 shows the results for the first type of queries. The first row shows results for Oldenburg and the second row for San Francisco. For both datasets, we show the results of random queries of different sizes, 1%, 10% and 20% in each dimension. Similar frameworks will be used to evaluate the other two types of queries. This is the same experimental setting used in [14].

In all the experiments our proposal outperforms both baselines. Just for small queries, 1% of the dimensions, the FNR-tree shows competitive results with our proposal. This is more evident in the largest network. The justification is the relative importance of the spatial part of the query with respect to the temporal part, which depends on the size of the network. Also important, our proposal shows better scalability on the number of objects moving through the network.

Figure 8 shows the results for range queries with larger temporal extent. In these experiments the temporal extent is always larger than the spatial extent, expanding the whole temporal dimension in the second and third column.

Results in this scenario are similar to the previous one, but the advantage of our proposal is even more obvious. Recall that our structure performs two rank operations in each independent set and then it just iterates over the results, which is very efficient. Finally, Figure 9 shows the results for time slice queries.

The analysis of these experiments is quite different from the previous ones, as the FNR-tree usually outperforms all the other approaches. There are two main reasons for this. First, large spatial queries lead querying many temporal indexes (all of them for the experiments in the last column). Second, most of these queries to temporal indexes produce empty results or very few results, which is expensive in our proposal. Each of these queries needs to perform the two rank operations in each independent set just to detect that there are no results to iterate through. Hence, this scenario represents the worst case for our proposal.

5 Conclusions

We have proposed a new data structure for trajectories of moving objects, which movement is constrained to a network. Our proposal is inspired by two-level indexes, such as the FNR-tree and MON-tree and, indeed, we use the same two-dimensional R-tree for the spatial dimension. Hence, the difference from previous solutions is in the temporal dimension. This is justified by our experimental evaluation showing that the spatial dimension requires negligible space compared with the temporal dimension. For this dimension, we propose a structure based on a decomposition on independent sets of intervals and the use of succinct data structures. Our experimental evaluation shows that the resulting structure is smaller than previous solutions, and also faster for a broad set of queries.

The interval intersection problem can be reduced to 2-sided range reporting [11], a problem for which efficient data structures have been successfully applied in LZ-indexes [3, 4]. As these structures are not adaptive to the number of independent interval sets, a combination of both approaches would be interesting as future work. Second, to handle larger datasets, it is necessary to improve construction time. Note, however, that we used larger datasets than those used in the evaluation of the FNR-tree. Third, some parts of the structure could be further optimized. We have observed that the distribution of the moving objects through the network is very skewed, which produces few temporal indexes storing many intervals and many indexes storing very few intervals. Hence, in order to use this index in practice, it is necessary to determine a threshold under which the intervals are just stored in an array and sequentially searched. Finally, bitmaps supporting append operations should be used to support dynamism.