Conformal Set-based Human-AI Complementarity with Multiple Experts

TL;DR

This paper introduces a greedy algorithm for selecting the most relevant human experts for each instance in a classification task, leveraging conformal prediction sets to improve decision support system performance.

Contribution

It presents a novel subset selection method for multiple experts using conformal sets, enhancing collaborative classification accuracy.

Findings

Greedy algorithm effectively identifies relevant experts for each instance.

Improved classification performance over naive expert selection methods.

Near-optimal expert subsets achieved in experiments with CIFAR-10H and ImageNet-16H.

Abstract

Decision support systems are designed to assist human experts in classification tasks by providing conformal prediction sets derived from a pre-trained model. This human-AI collaboration has demonstrated enhanced classification performance compared to using either the model or the expert independently. In this study, we focus on the selection of instance-specific experts from a pool of multiple human experts, contrasting it with existing research that typically focuses on single-expert scenarios. We characterize the conditions under which multiple experts can benefit from the conformal sets. With the insight that only certain experts may be relevant for each instance, we explore the problem of subset selection and introduce a greedy algorithm that utilizes conformal sets to identify the subset of expert predictions that will be used in classifying an instance. This approach is shown to yield better performance compared to naive methods for human subset selection. Based on real expert predictions from the CIFAR-10H and ImageNet-16H datasets, our simulation study indicates that our proposed greedy algorithm achieves near-optimal subsets, resulting in improved classification performance among multiple experts.