A Study on Evaluation Standard for Automatic Crack Detection Regard the Random Fractal

TL;DR

This paper identifies limitations of the traditional mAP standard for crack detection evaluation due to fractal features and proposes CovEval, a new standard that better aligns with visual performance and industrial needs.

Contribution

The paper introduces CovEval, a novel evaluation standard for crack detection that accounts for fractal features and improves scoring accuracy over mAP.

Findings

CovEval provides higher, more accurate scores for crack detectors.

Models achieve up to 95.8% Extended Recall with CovEval.

CovEval aligns better with visual inspection results.

Abstract

A reasonable evaluation standard underlies construction of effective deep learning models. However, we find in experiments that the automatic crack detectors based on deep learning are obviously underestimated by the widely used mean Average Precision (mAP) standard. This paper presents a study on the evaluation standard. It is clarified that the random fractal of crack disables the mAP standard, because the strict box matching in mAP calculation is unreasonable for the fractal feature. As a solution, a fractal-available evaluation standard named CovEval is proposed to correct the underestimation in crack detection. In CovEval, a different matching process based on the idea of covering box matching is adopted for this issue. In detail, Cover Area rate (CAr) is designed as a covering overlap, and a multi-match strategy is employed to release the one-to-one matching restriction in mAP. Extended Recall (XR), Extended Precision (XP) and Extended F-score (Fext) are defined for scoring the crack detectors. In experiments using several common frameworks for object detection, models get much higher scores in crack detection according to CovEval, which matches better with the visual performance. Moreover, based on faster R-CNN framework, we present a case study to optimize a crack detector based on CovEval standard. Recall (XR) of our best model achieves an industrial-level at 95.8, which implies that with reasonable standard for evaluation, the methods for object detection are with great potential for automatic industrial inspection.