A Systematic Evaluation of Generative Models on Tabular Transportation Data

TL;DR

This paper systematically evaluates various generative models for creating synthetic transportation data, highlighting their limitations and proposing new metrics to better assess their structural and privacy-preserving qualities.

Contribution

It introduces a novel graph-based metric and an improved privacy metric tailored for transportation data, and provides a comprehensive evaluation of existing models on NYC taxi data.

Findings

Existing models often underperform on transportation data.

The new graph metric reveals significant structural gaps.

Current models may not ensure adequate privacy protection.

Abstract

The sharing of large-scale transportation data is beneficial for transportation planning and policymaking. However, it also raises significant security and privacy concerns, as the data may include identifiable personal information, such as individuals' home locations. To address these concerns, synthetic data generation based on real transportation data offers a promising solution that allows privacy protection while potentially preserving data utility. Although there are various synthetic data generation techniques, they are often not tailored to the unique characteristics of transportation data, such as the inherent structure of transportation networks formed by all trips in the datasets. In this paper, we use New York City taxi data as a case study to conduct a systematic evaluation of the performance of widely used tabular data generative models. In addition to traditional metrics such as distribution similarity, coverage, and privacy preservation, we propose a novel graph-based metric tailored specifically for transportation data. This metric evaluates the similarity between real and synthetic transportation networks, providing potentially deeper insights into their structural and functional alignment. We also introduced an improved privacy metric to address the limitations of the commonly-used one. Our experimental results reveal that existing tabular data generative models often fail to perform as consistently as claimed in the literature, particularly when applied to transportation data use cases. Furthermore, our novel graph metric reveals a significant gap between synthetic and real data. This work underscores the potential need to develop generative models specifically tailored to take advantage of the unique characteristics of emerging domains, such as transportation.