Evaluating Generative Vehicle Trajectory Models for Traffic Intersection Dynamics

TL;DR

This paper develops and evaluates a comprehensive set of traffic engineering-specific metrics for generative vehicle trajectory models at intersections, highlighting discrepancies between low reconstruction errors and rule-breaking behaviors in simulated traffic.

Contribution

It introduces new evaluation metrics for generative traffic models that better reflect real-world safety and rule compliance concerns, beyond traditional reconstruction errors.

Findings

Models show low trajectory errors but break traffic rules in simulations.

New metrics reveal unsafe behaviors not captured by existing evaluation methods.

Online evaluation in microsimulation uncovers discrepancies between model accuracy and safety.

Abstract

Traffic Intersections are vital to urban road networks as they regulate the movement of people and goods. However, they are regions of conflicting trajectories and are prone to accidents. Deep Generative models of traffic dynamics at signalized intersections can greatly help traffic authorities better understand the efficiency and safety aspects. At present, models are evaluated on computational metrics that primarily look at trajectory reconstruction errors. They are not evaluated online in a `live' microsimulation scenario. Further, these metrics do not adequately consider traffic engineering-specific concerns such as red-light violations, unallowed stoppage, etc. In this work, we provide a comprehensive analytics tool to train, run, and evaluate models with metrics that give better insights into model performance from a traffic engineering point of view. We train a state-of-the-art multi-vehicle trajectory forecasting model on a large dataset collected by running a calibrated scenario of a real-world urban intersection. We then evaluate the performance of the prediction models, online in a microsimulator, under unseen traffic conditions. We show that despite using ideally-behaved trajectories as input, and achieving low trajectory reconstruction errors, the generated trajectories show behaviors that break traffic rules. We introduce new metrics to evaluate such undesired behaviors and present our results.