Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios

TL;DR

This paper introduces a structured benchmarking framework for evaluating human behavior prediction models in gap acceptance scenarios, revealing current models' unreliability in safety-critical situations.

Contribution

The paper develops a versatile evaluation framework for prediction models in traffic interactions, addressing dataset heterogeneity and error asymmetry, and applies it to assess state-of-the-art models.

Findings

Models are unreliable in safety-critical situations

The framework allows flexible evaluation of any model and metric

Current models need improvement for autonomous vehicle safety

Abstract

Autonomous vehicles currently suffer from a time-inefficient driving style caused by uncertainty about human behavior in traffic interactions. Accurate and reliable prediction models enabling more efficient trajectory planning could make autonomous vehicles more assertive in such interactions. However, the evaluation of such models is commonly oversimplistic, ignoring the asymmetric importance of prediction errors and the heterogeneity of the datasets used for testing. We examine the potential of recasting interactions between vehicles as gap acceptance scenarios and evaluating models in this structured environment. To that end, we develop a framework aiming to facilitate the evaluation of any model, by any metric, and in any scenario. We then apply this framework to state-of-the-art prediction models, which all show themselves to be unreliable in the most safety-critical situations.