Reconsidering utility: unveiling the limitations of synthetic mobility data generation algorithms in real-life scenarios

TL;DR

This paper critically evaluates the real-world applicability of five state-of-the-art synthetic mobility data generation models, revealing significant limitations in their ability to produce meaningful, accurate, and utility-preserving trip data, especially when incorporating privacy guarantees.

Contribution

The study provides a comprehensive real-world utility assessment of existing synthetic mobility models, highlighting their shortcomings in modeling trip sequences and traffic flow, and emphasizes the need for improved evaluation methods.

Findings

Three models maintain spatial distribution reasonably well.

One model fails to produce timely data; another generates excessive jumps.

All models struggle with trip length, traffic flow, and temporal data modeling.

Abstract

In recent years, there has been a surge in the development of models for the generation of synthetic mobility data. These models aim to facilitate the sharing of data while safeguarding privacy, all while ensuring high utility and flexibility regarding potential applications. However, current utility evaluation methods fail to fully account for real-life requirements. We evaluate the utility of five state-of-the-art synthesis approaches, each with and without the incorporation of differential privacy (DP) guarantees, in terms of real-world applicability. Specifically, we focus on so-called trip data that encode fine granular urban movements such as GPS-tracked taxi rides. Such data prove particularly valuable for downstream tasks at the road network level. Thus, our initial step involves appropriately map matching the synthetic data and subsequently comparing the resulting trips with those generated by the routing algorithm implemented in OpenStreetMap, which serves as an efficient and privacy-friendly baseline. Out of the five evaluated models, one fails to produce data within reasonable computation time and another generates too many jumps to meet the requirements for map matching. The remaining three models succeed to a certain degree in maintaining spatial distribution, one even with DP guarantees. However, all models struggle to produce meaningful sequences of geo-locations with reasonable trip lengths and to model traffic flow at intersections accurately. It is important to note that trip data encompasses various relevant characteristics beyond spatial distribution, such as temporal information, all of which are discarded by these models. Consequently, our results imply that current synthesis models fall short in their promise of high utility and flexibility.