Logical Assessment Formula and Its Principles for Evaluations with Inaccurate Ground-Truth Labels

TL;DR

This paper introduces the logical assessment formula (LAF), a new method for evaluating AI models when accurate ground-truth labels are unavailable, especially in challenging fields like medical image analysis.

Contribution

The paper proposes the LAF and its underlying principles, enabling logical evaluation of models with inaccurate ground-truth labels in complex scenarios.

Findings

LAF can evaluate models on difficult tasks with IAGTLs.

LAF can logically evaluate models on easier tasks with IAGTLs.

LAF acts similarly to traditional methods when AGTLs are available.

Abstract

Evaluations with accurate ground-truth labels (AGTLs) have been widely employed to assess predictive models for artificial intelligence applications. However, in some specific fields, such as medical histopathology whole slide image analysis, it is quite usual the situation that AGTLs are difficult to be precisely defined or even do not exist. To alleviate this situation, we propose logical assessment formula (LAF) and reveal its principles for evaluations with inaccurate ground-truth labels (IAGTLs) via logical reasoning under uncertainty. From the revealed principles of LAF, we summarize the practicability of LAF: 1) LAF can be applied for evaluations with IAGTLs on a more difficult task, able to act like usual strategies for evaluations with AGTLs reasonably; 2) LAF can be applied for evaluations with IAGTLs from the logical perspective on an easier task, unable to act like usual strategies for evaluations with AGTLs confidently.