Hierarchical Conformal Classification

TL;DR

Hierarchical conformal classification (HCC) extends conformal prediction by integrating class hierarchies, producing more informative and semantically meaningful prediction sets with guaranteed coverage across diverse data types.

Contribution

HCC introduces a novel hierarchical approach to conformal prediction, incorporating class structures into the prediction process while maintaining coverage guarantees.

Findings

HCC outperforms flat conformal methods on audio, image, and text benchmarks.

Hierarchical prediction sets are preferred by annotators over flat sets.

The approach reduces computational complexity by focusing on a smaller candidate solution set.

Abstract

Conformal prediction (CP) is a powerful framework for quantifying uncertainty in machine learning models, offering reliable predictions with finite-sample coverage guarantees. When applied to classification, CP produces a prediction set of possible labels that is guaranteed to contain the true label with high probability, regardless of the underlying classifier. However, standard CP treats classes as flat and unstructured, ignoring domain knowledge such as semantic relationships or hierarchical structure among class labels. This paper presents hierarchical conformal classification (HCC), an extension of CP that incorporates class hierarchies into both the structure and semantics of prediction sets. We formulate HCC as a constrained optimization problem whose solutions yield prediction sets composed of nodes at different levels of the hierarchy, while maintaining coverage guarantees. To address the combinatorial nature of the problem, we formally show that a much smaller, well-structured subset of candidate solutions suffices to ensure coverage while upholding optimality. An empirical evaluation on three new benchmarks consisting of audio, image, and text data highlights the advantages of our approach, and a user study shows that annotators significantly prefer hierarchical over flat prediction sets.