Towards Driving-Oriented Metric for Lane Detection Models

TL;DR

This paper introduces two new driving-oriented metrics for lane detection that better reflect autonomous driving requirements, and demonstrates that traditional metrics may not correlate well with real-world driving performance.

Contribution

It proposes E2E-LD and PSLD metrics tailored for autonomous driving, and empirically evaluates their effectiveness across multiple lane detection approaches and datasets.

Findings

Traditional metrics have negative correlation with driving-oriented metrics.

Recent improvements may overfit to conventional metrics without improving real-world safety.

Driving-oriented metrics provide more meaningful evaluation for autonomous driving applications.

Abstract

After the 2017 TuSimple Lane Detection Challenge, its dataset and evaluation based on accuracy and F1 score have become the de facto standard to measure the performance of lane detection methods. While they have played a major role in improving the performance of lane detection methods, the validity of this evaluation method in downstream tasks has not been adequately researched. In this study, we design 2 new driving-oriented metrics for lane detection: End-to-End Lateral Deviation metric (E2E-LD) is directly formulated based on the requirements of autonomous driving, a core downstream task of lane detection; Per-frame Simulated Lateral Deviation metric (PSLD) is a lightweight surrogate metric of E2E-LD. To evaluate the validity of the metrics, we conduct a large-scale empirical study with 4 major types of lane detection approaches on the TuSimple dataset and our newly constructed dataset Comma2k19-LD. Our results show that the conventional metrics have strongly negative correlations ($\leq$-0.55) with E2E-LD, meaning that some recent improvements purely targeting the conventional metrics may not have led to meaningful improvements in autonomous driving, but rather may actually have made it worse by overfitting to the conventional metrics. As autonomous driving is a security/safety-critical system, the underestimation of robustness hinders the sound development of practical lane detection models. We hope that our study will help the community achieve more downstream task-aware evaluations for lane detection.