The Effectiveness of Edge Detection Evaluation Metrics for Automated Coastline Detection

TL;DR

This study evaluates the effectiveness of various image quality metrics for automated coastline detection, finding that FOM outperforms traditional metrics like RMSE, PSNR, and SSIM in selecting optimal edge detection thresholds.

Contribution

The paper demonstrates that FOM is a more reliable metric for coastline edge detection, providing a reformulation of common metrics in terms of confusion matrix measures.

Findings

FOM correctly selects the best threshold 92.6% of the time.

RMSE, PSNR, and SSIM perform poorly, with less than 12% success.

Reformulation explains why traditional metrics are ineffective for edge detection evaluation.

Abstract

We analyse the effectiveness of RMSE, PSNR, SSIM and FOM for evaluating edge detection algorithms used for automated coastline detection. Typically, the accuracy of detected coastlines is assessed visually. This can be impractical on a large scale leading to the need for objective evaluation metrics. Hence, we conduct an experiment to find reliable metrics. We apply Canny edge detection to 95 coastline satellite images across 49 testing locations. We vary the Hysteresis thresholds and compare metric values to a visual analysis of detected edges. We found that FOM was the most reliable metric for selecting the best threshold. It could select a better threshold 92.6% of the time and the best threshold 66.3% of the time. This is compared RMSE, PSNR and SSIM which could select the best threshold 6.3%, 6.3% and 11.6% of the time respectively. We provide a reason for these results by reformulating RMSE, PSNR and SSIM in terms of confusion matrix measures. This suggests these metrics not only fail for this experiment but are not useful for evaluating edge detection in general.