Understanding implementation pitfalls of distance-based metrics for image segmentation

TL;DR

This paper critically examines the implementation differences of distance-based metrics like Hausdorff distance in image segmentation, revealing significant discrepancies that affect benchmarking and clinical applications.

Contribution

It provides a systematic analysis of 11 open-source tools, highlighting implementation discrepancies and their impact on segmentation validation.

Findings

Deviations in Hausdorff distance exceeding 100 mm observed

Statistically significant differences between tools identified

Implementation choices can lead to false impressions of performance improvements

Abstract

Distance-based metrics, such as the Hausdorff distance (HD), are widely used to validate segmentation performance in (bio)medical imaging. However, their implementation is complex, and critical differences across open-source tools remain largely unrecognized by the community. These discrepancies undermine benchmarking efforts, introduce bias in biomarker calculations, and potentially distort medical device development and clinical commissioning. In this study, we systematically dissect 11 open-source tools that implement distance-based metric computation by performing both a conceptual analysis of their computational steps and an empirical analysis on representative two- and three-dimensional image datasets. Alarmingly, we observed deviations in HD exceeding 100 mm and identified multiple statistically significant differences between tools - demonstrating that statistically significant improvements on the same set of segmentations can be achieved simply by selecting a particular implementation. These findings cast doubts on the validity of prior comparisons of results across studies without accounting for the differences in metric implementations. To address this, we provide practical recommendations for tool selection; additionally, our conceptual analysis informs about the future evolution of implementing open-source tools.