Evaluating Human Trajectory Prediction with Metamorphic Testing

TL;DR

This paper applies metamorphic testing to evaluate human trajectory prediction models, addressing the challenge of lacking clear correctness criteria by exploiting behavioral invariants and introducing a Wasserstein Violation Criterion.

Contribution

It introduces a novel application of metamorphic testing to stochastic human trajectory prediction and proposes a Wasserstein-based statistical assessment method.

Findings

Metamorphic testing effectively evaluates trajectory prediction models.

The Wasserstein Violation Criterion detects violations of invariants.

Method handles the inherent uncertainty in human behavior prediction.

Abstract

The prediction of human trajectories is important for planning in autonomous systems that act in the real world, e.g. automated driving or mobile robots. Human trajectory prediction is a noisy process, and no prediction does precisely match any future trajectory. It is therefore approached as a stochastic problem, where the goal is to minimise the error between the true and the predicted trajectory. In this work, we explore the application of metamorphic testing for human trajectory prediction. Metamorphic testing is designed to handle unclear or missing test oracles. It is well-designed for human trajectory prediction, where there is no clear criterion of correct or incorrect human behaviour. Metamorphic relations rely on transformations over source test cases and exploit invariants. A setting well-designed for human trajectory prediction where there are many symmetries of expected human behaviour under variations of the input, e.g. mirroring and rescaling of the input data. We discuss how metamorphic testing can be applied to stochastic human trajectory prediction and introduce the Wasserstein Violation Criterion to statistically assess whether a follow-up test case violates a label-preserving metamorphic relation.