Calibration improves detection of mislabeled examples

TL;DR

This paper demonstrates that calibrating the base model in mislabel detection significantly enhances the accuracy and robustness of identifying mislabeled data, offering a practical solution for industrial machine learning systems.

Contribution

The study shows that calibration methods improve the effectiveness of mislabel detection by enhancing the trust scores derived from the base model.

Findings

Calibration improves detection accuracy

Calibration increases robustness against mislabeled data

Empirical results confirm practical benefits

Abstract

Mislabeled data is a pervasive issue that undermines the performance of machine learning systems in real-world applications. An effective approach to mitigate this problem is to detect mislabeled instances and subject them to special treatment, such as filtering or relabeling. Automatic mislabeling detection methods typically rely on training a base machine learning model and then probing it for each instance to obtain a trust score that each provided label is genuine or incorrect. The properties of this base model are thus of paramount importance. In this paper, we investigate the impact of calibrating this model. Our empirical results show that using calibration methods improves the accuracy and robustness of mislabeled instance detection, providing a practical and effective solution for industrial applications.