Towards Reliable Evaluation of Road Network Reconstructions

TL;DR

This paper identifies flaws in existing road network reconstruction metrics and proposes three new, more consistent evaluation metrics validated on synthetic and real data to improve future assessments.

Contribution

The paper introduces three novel metrics for evaluating road network reconstructions that address inconsistencies in existing measures.

Findings

Existing metrics are inconsistent and insensitive to certain errors.

New metrics show higher consistency across different error classes.

Validated on synthetic and real datasets, these metrics improve evaluation reliability.

Abstract

Existing performance measures rank delineation algorithms inconsistently, which makes it difficult to decide which one is best in any given situation. We show that these inconsistencies stem from design flaws that make the metrics insensitive to whole classes of errors. To provide more reliable evaluation, we design three new metrics that are far more consistent even though they use very different approaches to comparing ground-truth and reconstructed road networks. We use both synthetic and real data to demonstrate this and advocate the use of these corrected metrics as a tool to gauge future progress.