A Natural-language-based Visual Query Approach of Uncertain Human Trajectories

TL;DR

This paper introduces a natural-language-based visual query system for uncertain human trajectory data, enabling effective exploration and analysis despite data inaccuracies and uncertainty.

Contribution

It presents a novel approach that extracts spatial-temporal constraints from natural language and encodes uncertain trajectories using semantic POI information.

Findings

Effective querying over uncertain trajectory data demonstrated on real datasets

Semantic encoding improves trajectory data retrieval and visualization

Supports natural language interaction for complex spatial-temporal queries

Abstract

Visual querying is essential for interactively exploring massive trajectory data. However, the data uncertainty imposes profound challenges to fulfill advanced analytics requirements. On the one hand, many underlying data does not contain accurate geographic coordinates, e.g., positions of a mobile phone only refer to the regions (i.e., mobile cell stations) in which it resides, instead of accurate GPS coordinates. On the other hand, domain experts and general users prefer a natural way, such as using a natural language sentence, to access and analyze massive movement data. In this paper, we propose a visual analytics approach that can extract spatial-temporal constraints from a textual sentence and support an effective query method over uncertain mobile trajectory data. It is built up on encoding massive, spatially uncertain trajectories by the semantic information of the POIs and regions covered by them, and then storing the trajectory documents in text database with an effective indexing scheme. The visual interface facilitates query condition specification, situation-aware visualization, and semantic exploration of large trajectory data. Usage scenarios on real-world human mobility datasets demonstrate the effectiveness of our approach.

Tables

Table 1. Table 1: Examples of the dictionary of temporal constraints

oX[l] X[2.5l]   Query Syntax	Description
morning	From 6:00 to 9:00
noon	From 11:00 to 14:00
evening	From 18:00 to 24:00
during (T)	Time conjunction, where T is a time period
A after B	Trajectory go to place ’B’ after place ‘A’.
A before B	Trajectory go to place ’A’ before place ‘B’

Table 2. Table 2: Query Performance for Three Tasks

oX[2l] X[l] X[l] X[l]   Task	Task1 (sec)	Task2 (sec)	Task3 (sec)
Data Access Time	2.07	13.48	4.35

Equations

$$sim(w_1,w_2)=\frac{\overrightarrow{w_1}\cdot \overrightarrow{w_2}}{\left|\overrightarrow{w_1}\right|\cdot \left|\overrightarrow{w_2}\right|},$$

$$Score(D,W)=\sum _{i=1}^{n}R(w_i,D),$$

$$R(w_i,D)=IDF(w_i)\cdot \frac{TF(w_i)\cdot (k+1)}{TF(w_i)+k\cdot (1-b+b\cdot \frac{|D|}{avgdl})}.$$

$$IDF(w_i)=log\frac{m-M(w_i)+0.5}{M(w_i)+0.5},$$

$$Scor{e}^{{}^{\prime }}(D,W)=\alpha \cdot Score(D,W)+\beta \cdot si{m}_{topic}(\overrightarrow{T_L},\overrightarrow{T_I}),$$

Full text

\teaser

(a) The query condition specification view. (b) The relevance tree with the spatial keyword ‘tourist attractions’. (c) The drop-down menu for changing the type of the input keyword. (d) The semantics view shows that the major region functional topic is ‘residential related’. (e) The map view shows that the queried trajectories are mainly distributed in the northwest (named ‘Jiangxin island’), the middle, and the east (named ‘Wuhua building’) of the city. (f) Most trajectories land the island from its east through ferry. (g) The region functional topics. (h) The rendering parameter widget. (i) The temporal graph view. (j) The detail result view of the queried trajectories. (k) Detail study of urban areas.

Introduction

A common task in analyzing massive trajectory data is to query the trajectories with given spatiotemporal conditions. This has been proven useful to improve people’s life quality [49], urban planning [72], and real-time monitoring [74, 67, 47]. A crucial problem in this process is to help data analysts express their query requirements intuitively and effectively, where interactive visual interfaces [16, 80] can play an important role. However, specifying complex conditions in spatial and temporal dimensions is neither natural nor intuitive for domain users and practitioners, which can easily hinder their intention of utilizing the systems.

Using natural language input is undoubtedly a preferred way to express query conditions, where analyzers can naturally use location names (e.g., Golden Gate Bridge), functional categories (e.g., education areas, residential areas), and time descriptions (e.g., morning) to filter massive trajectories. The “textualization-and-query” scheme has been applied by querying geospatial data after externalizing locations with contextual geo-information (a.k.a., textualization). The annotated trajectory data provides knowledge-based context [65] which dramatically enriches the raw data [2, 72, 58, 32]. This scheme is used in several visual analytics systems for studying taxi trajectories [17, 1] and human mobility patterns [75]. Most existing methods assume that the trajectories contain accurate geo-locations. However, many real-world trajectory datasets incorporate spatial uncertainty due to (1) the inaccuracy and measurement error of sensors [27]. GPS samples can divert from real positions while some samples along a trajectory may be missing; and (2) the location privacy protection [21]. Data providers may hide an accurate point of longitude and latitude with a spatial unit (e.g., street, region) it resides. Figure 1 illustrates human trajectories sampled by mobile phones, where each data point only tells us the base station that the phone is connected to. The dashed line denotes a real human trace which is not achievable, while the solid line links the stations to form a trajectory that approximates the real trace. The spatially uncertain trajectories have been proved essential in many application scenarios [4] such as identification of commute patterns [67, 80], analyzing people’s activities [14, 36], and spatial planning [44]. Similar to certain trajectories, the analytical goal is to quickly retrieve the trajectories based on spatiotemporal conditions and then discover patterns from the query results. However, spatially uncertain trajectories make it hard to access the data by specifying accurate locations. Therefore, querying them by a descriptive sentence such as “trajectories passing a school” becomes necessary and useful for analyzers. Existing textualization and query methods are not directly applicable since the sampling locations are not mapped to a fixed street name or POI (Point of Interest, which is a specific point location, e.g., hotel.), which motivates us to develop new approach that integrates natural language based query with uncertain trajectory data.

In this paper, we develop a new visual analytics system to fill the gap, which enables users to query spatially uncertain human trajectories with natural language sentences. For example, an urban planner can specify: ‘Query human trajectories passed through tourist attractions in the morning’ for discovering mobility patterns of tourists and improving public services. The system further provides a set of visual representations to help users easily study the returned trajectories. Our approach integrates high-level descriptive languages of locations and time. A trajectory query engine is developed to address two major challenges: the spatial uncertainty and inaccurate descriptive language. In this paper, we use the mobile phone trajectory data to describe our work, which can also be extended to handle other types of uncertain human trajectories (Sec. 5).

First, we understand and extract the conditions from a natural language query sentence. The spatiotemporal constraints are not given accurately. A location name often includes entry error [27] and have vague meanings. For example, ‘school, college, campus, and university’ may all be linked to one given word ‘school’. We address the challenge by hiring a natural language lexical analyzer to split and encode spatial and temporal expressions, and then developing a query condition specification tool to visually identify and refine relevant words, where a word embedding model is trained to discover word relevance.

Second, we query trajectories by the extracted conditions. First, the city space is subdivided by small possible spatial regions (PSR) to accommodate inaccurate sample points. Second, the trajectories are converted to documents containing POI information in the regions and then stored in a database with a special indexing scheme. Third, top-K POIs matching the input conditions are found by a probabilistic retrieval algorithm (Okapi BM25 [59]), which is further enhanced by utilizing the functional topics of regional POIs discovered by Latent Dirichlet Allocation (LDA [10]) topic modeling. Eventually, the top-K POIs are used to extract relevant trajectories in the database.

Furthermore, once a group of trajectories is acquired, we further rank them by a relevance score computed for each trajectory. We present an ordered list of the results which helps users efficiently investigate the trajectories that meet their criteria.

A visual analytics system is built up with (1) a query condition specification view to specify a query and visually adjust query conditions. (2) A map view to visualize trajectories within geo-context; and (3) a set of visualizations including a temporal graph view, a semantics view, and a detail result view for examining the query results.

Our contributions are summarized as follows:

• •

An efficient query engine of spatially uncertain human trajectories which is built upon natural language queries and textualization of the trajectories.

• •

A multi-faceted visual interface designed for interactively specifying semantic query conditions and investigating retrieved uncertain trajectories.

• •

Two real-world usage scenarios based on massive smartphone based human trajectories that demonstrate the usefulness of our approach.

1 Related Work

The goal of visual query is to retrieve and study trajectories by addressing the challenges of the large data volume and the diversity of query tasks [31, 50]. Here we introduce relevant literature on query conditions, trajectory data queries, and semantic-based visual exploration of spatial trajectory data.

1.1 Query Condition Specification

Retrieving trajectories via programming languages [78] is widely used in data analysis systems designed for programmers instead of casual users. Recently, visual query technology has been proven to be useful in helping users express their query requirements, such as defining spatial constraints on map [80, 66], specifying Origin-Destination (OD) query [26], and performing road navigation [46]. Visual query systems also allow users to modify their query input by well-designed interactions. For example, VAUD [16] helps users interactively construct and optimize data query conditions. VESPa [30] employs a sketch-based interface for users to express, check, and refine hypotheses.

However, for complex data query tasks, users often need to conduct cumbersome interactions to refine the conditions which are neither natural nor intuitive. Using natural language in visual queries has been shown as a natural way for user interaction. Alvares et al. [2] propose a generic model to enrich trajectories with semantic geographical information and support semantic queries and analysis. SemanticTraj [1] fuses GPS sampling points of trajectories with their street and POI semantics and enables querying via textual input of street/POI names. However, the work does not support users to specify a fuzzy yet more natural query sentence (e.g., ‘Query trajectories of students’). Our approach overcomes this problem by extracting query constraints from natural language sentences and supporting visual condition inspection and optimization.

1.2 Efficient Trajectory Queries

Spatial indexing is widely used for trajectory data management and supporting effective trajectory retrieval. Feng et al. [25] summarize seven kinds of queries, which mainly focuses on two categories [77]: Range queries and K-Nearest-Neighborhoods (KNN) queries.

Range queries refer to filtering the trajectories falling into a spatial region. The tree-based index [31] is one of the most popular indexing methods to support it. Mokbel and Chen et al. [52, 13] review and evaluate the performance of existing spatial-temporal access methods. The space-time cube (STC) [35] enables fast indexing and access by dividing the space-time space into uniformly sampled grids. Extended variants of the STC (e.g., Nanocubes [43]) are widely employed for supporting multivariate spatial-temporal data exploration [40, 53], aggregation [3, 11], and interactive analysis [39, 6, 8]. KNN-based queries refer to retrieving top K trajectories which have the minimum distance to given locations. R-tree [29, 20], KD-Tree [29, 20] and their extensions (e.g.,  [61, 56, 54, 79]) are widely used to improve the performance of KNN-based queries. SETI [12] indexes spatial and temporal dimensions separately by a two-level index structure. TrajStore [19] divides trajectories into subtrajectories and stores them geographically and temporally near each other in the same disk block. While necessary, the above indexing schemas focus on accessing data in spatial-temporal dimensions. In this paper, our query engine is built upon an efficient documentation-indexing scheme for the fast query of uncertain trajectories through a simple text-based input.

1.3 Semantic Visual Exploration of Trajectories

There are a large number of studies on visual analysis of trajectory data. We refer the readers to recent literature surveys for more details  [6, 15]. Existing works mainly focus on displaying spatial-temporal information of trajectories on a map and visual charts. However, human mobility data [64] are massive and have complex semantics. Understanding the semantics which contain enriched knowledge (e.g., behaviors of moving objects.) has been an important research task. A survey [55] summarizes the semantic enrichment, knowledge extraction, and mobility analysis approaches for the trajectory data. Many approaches use data fusing techniques to supplement trajectories from multiple domain data [62]. The POI data is often used to discover different and temporally changing functions of urban regions [72, 37] which can further enrich the trajectory data. Similarly, Zeng et al. [75] characterize population movement by the subway transportation data and related POIs for individual behavior investigation. Moreover, by reformulating trajectories into an appropriate semantic form  [17], the LDA topic models are utilized to extract implicit themes and visualize them.

To depict the spatiotemporal attributes of the trajectories, the geographical context information is incorporated into the trajectory visualization. For instance, a node-link graph  [33] is constructed and employed to study relations among different regions. The trajectory flow from OD movement data is encoded with a spatiotemporal abstraction to reveal patterns and trends of mass mobility [5]. Andrienko et al. [7] use a state transition graph to display the mobility behaviors where each state represents a semantic category of location (e.g., shop). Other representative works are dedicated to the exploration of stop points [14, 71, 73, 69], trajectory lines [45, 70], and regions [41, 74]. Most of these works focus on trajectories with accurate geo-location. Our approach instead seeks to represent and visualize semantics from spatially uncertain trajectory data. The intention is that we should allow users to intuitively use the geographical semantics and to achieve an iterative trajectory data exploration process.

2 National Language Based Query Engine

To support querying uncertain trajectories with high-level descriptive language, the engine is built up on several modules including: (1) an uncertain trajectory documentation and indexing module to represent and manage uncertain trajectories; (2) a constraint extraction module to process the input sentence and form specific spatial-temporal conditions; and (3) a relevance quantification method that computes the relevance scores of uncertain trajectories to the given conditions.

2.1 Uncertain Trajectory Documentation and Indexing

An uncertain trajectory has a series of inaccuracy locations instead of accurate longitudes and latitudes. Each such location typically can be considered residing in a possible spatial region (PSR). For example, an inaccurate GPS point may locate in a circular region with a radius of dozens of meters. A human mobile sample point may reside in the close neighborhood of a base station (Figure 1). We then design and implement an uncertain trajectory documentation and indexing mechanism based on the PSR.

Using human mobile trajectory data as an example, given the distribution of base stations, the urban space can be partitioned into PSR regions. As shown in Figure 1, a Voronoi partitioning [9] algorithm yields a set of disjoint Voronoi cells of a city, which can be considered as PSRs of the inaccurate trajectory points inside the city. Each city POI (Point of Interest) uniquely belongs to a certain PSR region, and each region contains multiple POIs.

Documentation: An uncertain human trajectory is enriched with its contextual information and transformed into a text formulation. As shown in Figure 2, each trajectory data point contains a timestamp, its geographic PSR information (including the base station ID, latitude and longitude), and its semantic information which are the names, types, and descriptions of the POIs inside its PSR. Then this information of all sample points on a trajectory forms a trajectory document. Following this process, all trajectories are converted into trajectory documents and stored into a text database.

Please note that for different types of uncertain trajectory data, the PSRs can be computed in other ways, but the trajectory documents can be generated in the same way using the POIs inside the PSRs.

Indexing: We propose a temporal-textual-trajectory index to support efficient semantic query. First, we divide the collection of trajectory documents into multiple data partitions based on a time interval, that is, trajectories staying in the same time window are fetched and stored into the same data partition. Then, trajectory documents in the same partition are stored into adjacent disk blocks to save disk I/O. Next, we build an inverted index for trajectory documents in the same data partition. This inverted index stores the IDs of trajectories whose documents contain the keywords of POIs names and types. Finally, we add the timestamps used for data partitions to generate a temporal-textual-trajectory index.

Retrieval: Given spatial and temporal query conditions, such as querying trajectories passing ‘Central Park’ during ‘1:00 $pm$ to 2:00 $pm$’. We first fetch the data partitions that meet the temporal constraint. Because data partitions are sorted by time, a binary search is used to speed up this searching whose runtime overhead is $O(\log p)$, where $p$ is the total number of data partitions. Then, for each qualified data partition, the inverted index identifies those trajectory documents that contain the keyword ‘Central Park’ which involves $O(1)$ runtime cost for each partition. Our experiments show that this indexing scheme can accelerate trajectory retrieval with less latency for real-time exploration and visualization.

2.2 Natural Language Processing and Constraints Extraction

When natural language sentences are used to query the trajectory documents, the key topic is to compute the relevance scores (i.e., similarities) between an input sentence and trajectories. The first step is to analyze the sentence and extract important words as query conditions. Given a sentence S, it generally contains the temporal and/or spatial constraint words and their conjunctions. A temporal-constraint word describes a time limitation such as a date or a descriptive word (e.g., morning). A spatial-constraint word describes semantic location information such as a POI name or category. Our query engine extracts the temporal and spatial constraints from the given sentence respectively.

The THU Lexical Analyzer (THULAC) [42] is employed to split a query sentence into words. THULAC is a word segmentation model and can identify whether a word is a conjunction, a noun or a temporal-related word. The words are automatically divided into conjunctions, temporal-constraint words, and spatial-constraint words.

Temporal Constraints: The temporal-constraint words are directly mapped to a time period or a time stamp. We pre-define several descriptive temporal words and conjunctions to denote temporal constraints, as shown in Table 2.2. The mapping from a word to a time window, such as ‘morning’ to ‘from 6 am to 9 am’, is arbitrarily defined which is easily adjustable by users. This table only enumerates some commonly used words which can be further extended (but may need to change the indexing scheme and query algorithm).