Bias-Aware Mislabeling Detection via Decoupled Confident Learning

TL;DR

This paper introduces DeCoLe, a machine learning framework designed to detect mislabeled data affected by bias, improving data quality especially in sensitive domains like hate speech detection.

Contribution

The paper presents DeCoLe, a novel bias-aware mislabeling detection method with theoretical justification and empirical validation in hate speech detection.

Findings

DeCoLe outperforms existing label error detection methods.

DeCoLe effectively detects bias-related mislabels in datasets.

The framework can be integrated into organizational data management.

Abstract

Reliable data is a cornerstone of modern organizational systems. A notable data integrity challenge stems from label bias, which refers to systematic errors in a label, a covariate that is central to a quantitative analysis, such that its quality differs across social groups. This type of bias has been conceptually and empirically explored and is widely recognized as a pressing issue across critical domains. However, effective methodologies for addressing it remain scarce. In this work, we propose Decoupled Confident Learning (DeCoLe), a principled machine learning based framework specifically designed to detect mislabeled instances in datasets affected by label bias, enabling bias aware mislabelling detection and facilitating data quality improvement. We theoretically justify the effectiveness of DeCoLe and evaluate its performance in the impactful context of hate speech detection, a domain where label bias is a well documented challenge. Empirical results demonstrate that DeCoLe excels at bias aware mislabeling detection, consistently outperforming alternative approaches for label error detection. Our work identifies and addresses the challenge of bias aware mislabeling detection and offers guidance on how DeCoLe can be integrated into organizational data management practices as a powerful tool to enhance data reliability.