Beelines: Motion Prediction Metrics for Self-Driving Safety and Comfort

TL;DR

This paper introduces new motion prediction metrics tailored for self-driving cars that better correlate with safety and comfort, improving evaluation efficiency over traditional displacement errors.

Contribution

It proposes two novel metrics focused on safety and comfort, addressing the limitations of existing displacement error metrics in self-driving system evaluation.

Findings

Safety metric outperforms displacement error in identifying unsafe events

Metrics show better correlation with system-level safety and comfort

Simulation results validate the effectiveness of proposed metrics

Abstract

The commonly used metrics for motion prediction do not correlate well with a self-driving vehicle's system-level performance. The most common metrics are average displacement error (ADE) and final displacement error (FDE), which omit many features, making them poor self-driving performance indicators. Since high-fidelity simulations and track testing can be resource-intensive, the use of prediction metrics better correlated with full-system behavior allows for swifter iteration cycles. In this paper, we offer a conceptual framework for prediction evaluation highly specific to self-driving. We propose two complementary metrics that quantify the effects of motion prediction on safety (related to recall) and comfort (related to precision). Using a simulator, we demonstrate that our safety metric has a significantly better signal-to-noise ratio than displacement error in identifying unsafe events.