Distribution-Free Prediction Sets for Two-Layer Hierarchical Models

TL;DR

This paper extends conformal prediction methods to two-layer hierarchical models, enabling distribution-free prediction sets in complex data structures with multiple groups, balancing coverage guarantees and set size.

Contribution

It introduces new hierarchical conformal prediction techniques, including CDF pooling and subsampling methods, for both supervised and unsupervised settings.

Findings

CDF pooling balances coverage and set size in large samples.

Repeated subsampling provides stronger coverage guarantees.

Methods outperform traditional conformal prediction in hierarchical data.

Abstract

We consider the problem of constructing distribution-free prediction sets for data from two-layer hierarchical distributions. For iid data, prediction sets can be constructed using the method of conformal prediction. The validity of conformal prediction hinges on the exchangeability of the data, which does not hold when groups of observations come from distinct distributions, such as multiple observations on each patient in a medical database. We extend conformal methods to this hierarchical setting. We develop CDF pooling, single subsampling, and repeated subsampling approaches to construct prediction sets in unsupervised and supervised settings. We compare these approaches in terms of coverage and average set size. If asymptotic coverage is acceptable, we recommend CDF pooling for its balance between empirical coverage and average set size. If we desire coverage guarantees, then we recommend the repeated subsampling approach.