Comparison of automated crater catalogs for Mars from Benedix et al. (2020) and Lee and Hogan (2021)

TL;DR

This paper compares two automated crater catalogs for Mars, revealing how performance metrics and geographic factors influence perceived accuracy and highlighting the importance of validation methods for neural network-based planetary mapping.

Contribution

It provides a detailed comparison of two crater detection algorithms, identifies issues affecting performance at higher latitudes, and emphasizes validation strategies for neural network applications in planetary science.

Findings

Performance varies with metrics used for comparison.

Latitude affects crater detection accuracy, with higher latitudes showing performance loss.

Image projection issues may cause geographic biases in crater catalogs.

Abstract

Crater mapping using neural networks and other automated methods has increased recently with automated Crater Detection Algorithms (CDAs) applied to planetary bodies throughout the solar system. A recent publication by Benedix et al. (2020) showed high performance at small scales compared to similar automated CDAs but with a net positive diameter bias in many crater candidates. I compare the publicly available catalogs from Benedix et al. (2020) and Lee & Hogan (2021) and show that the reported performance is sensitive to the metrics used to test the catalogs. I show how the more permissive comparison methods indicate a higher CDA performance by allowing worse candidate craters to match ground-truth craters. I show that the Benedix et al. (2020) catalog has a substantial performance loss with increasing latitude and identify an image projection issue that might cause this loss. Finally, I suggest future applications of neural networks in generating large scientific datasets be validated using secondary networks with independent data sources or training methods.