Conformalized Semi-supervised Random Forest for Classification and Abnormality Detection

TL;DR

This paper introduces CSForest, a novel semi-supervised conformal prediction method for classification that effectively detects outliers and adapts to distribution shifts using unlabeled test data.

Contribution

The paper proposes CSForest, combining conformalization and semi-supervised learning, to improve outlier detection and prediction accuracy under distribution shifts.

Findings

CSForest accurately detects outliers in test data.

It maintains high prediction coverage for inlier classes.

Performs well across various datasets and distribution changes.

Abstract

The Random Forests classifier, a widely utilized off-the-shelf classification tool, assumes training and test samples come from the same distribution as other standard classifiers. However, in safety-critical scenarios like medical diagnosis and network attack detection, discrepancies between the training and test sets, including the potential presence of novel outlier samples not appearing during training, can pose significant challenges. To address this problem, we introduce the Conformalized Semi-Supervised Random Forest (CSForest), which couples the conformalization technique Jackknife+aB with semi-supervised tree ensembles to construct a set-valued prediction $C(x)$. Instead of optimizing over the training distribution, CSForest employs unlabeled test samples to enhance accuracy and flag unseen outliers by generating an empty set. Theoretically, we establish CSForest to cover true labels for previously observed inlier classes under arbitrarily label-shift in the test data. We compare CSForest with state-of-the-art methods using synthetic examples and various real-world datasets, under different types of distribution changes in the test domain. Our results highlight CSForest's effective prediction of inliers and its ability to detect outlier samples unique to the test data. In addition, CSForest shows persistently good performance as the sizes of the training and test sets vary. Codes of CSForest are available at https://github.com/yujinhan98/CSForest.