Multi-label Classification under Uncertainty: A Tree-based Conformal Prediction Approach

TL;DR

This paper introduces a novel tree-based conformal prediction method for multi-label classification that guarantees valid coverage and controls error rates, demonstrated through simulations and real data analysis.

Contribution

It develops a hierarchical tree-based conformal prediction approach with error rate control for multi-label classification, a novel combination not previously explored.

Findings

Effective in controlling family-wise error rate

Provides valid coverage guarantees

Outperforms existing conformal methods in experiments

Abstract

Multi-label classification is a common challenge in various machine learning applications, where a single data instance can be associated with multiple classes simultaneously. The current paper proposes a novel tree-based method for multi-label classification using conformal prediction and multiple hypothesis testing. The proposed method employs hierarchical clustering with labelsets to develop a hierarchical tree, which is then formulated as a multiple-testing problem with a hierarchical structure. The split-conformal prediction method is used to obtain marginal conformal $p$-values for each tested hypothesis, and two \textit{hierarchical testing procedures} are developed based on marginal conformal $p$-values, including a hierarchical Bonferroni procedure and its modification for controlling the family-wise error rate. The prediction sets are thus formed based on the testing outcomes of these two procedures. We establish a theoretical guarantee of valid coverage for the prediction sets through proven family-wise error rate control of those two procedures. We demonstrate the effectiveness of our method in a simulation study and two real data analysis compared to other conformal methods for multi-label classification.