Enhancing Evaluation Methods for Infrared Small-Target Detection in Real-world Scenarios

TL;DR

This paper critically evaluates existing IR small-target detection metrics, proposes new evaluation metrics aligned with real-world needs, and demonstrates their effectiveness through comparative analysis of five algorithms.

Contribution

It introduces novel evaluation metrics for infrared small-target detection that better reflect real-world performance conditions.

Findings

Existing metrics have limitations in real-world scenarios.

New metrics provide more consistent and meaningful assessments.

Evaluation of five algorithms shows improved evaluation accuracy.

Abstract

Infrared small target detection (IRSTD) poses a significant challenge in the field of computer vision. While substantial efforts have been made over the past two decades to improve the detection capabilities of IRSTD algorithms, there has been a lack of extensive investigation into the evaluation metrics used for assessing their performance. In this paper, we employ a systematic approach to address this issue by first evaluating the effectiveness of existing metrics and then proposing new metrics to overcome the limitations of conventional ones. To achieve this, we carefully analyze the necessary conditions for successful detection and identify the shortcomings of current evaluation metrics, including both pre-thresholding and post-thresholding metrics. We then introduce new metrics that are designed to align with the requirements of real-world systems. Furthermore, we utilize these newly proposed metrics to compare and evaluate the performance of five widely recognized small infrared target detection algorithms. The results demonstrate that the new metrics provide consistent and meaningful quantitative assessments, aligning with qualitative observations.