Effects of label noise on the classification of outlier observations

TL;DR

This paper examines how label noise affects the performance of the BCOPS conformal prediction algorithm in classifying outliers, revealing that even small noise levels can significantly impact robustness and prediction accuracy.

Contribution

It provides the first analysis of label noise impact on BCOPS, demonstrating the algorithm's sensitivity and robustness issues in noisy classification scenarios.

Findings

Small label noise significantly reduces prediction accuracy.

Noise increases the abstention rate for outliers.

Model robustness is compromised by even minimal label noise.

Abstract

This study investigates the impact of adding noise to the training set classes in classification tasks using the BCOPS algorithm (Balanced and Conformal Optimized Prediction Sets), proposed by Guan & Tibshirani (2022). The BCOPS algorithm is an application of conformal prediction combined with a machine learning method to construct prediction sets such that the probability of the true class being included in the prediction set for a test observation meets a specified coverage guarantee. An observation is considered an outlier if its true class is not present in the training set. The study employs both synthetic and real datasets and conducts experiments to evaluate the prediction abstention rate for outlier observations and the model's robustness in this previously untested scenario. The results indicate that the addition of noise, even in small amounts, can have a significant effect on model performance.