Efficient Ensemble Selection from Binary and Pairwise Feedback

TL;DR

This paper addresses the challenge of selecting high-performing AI ensembles efficiently using binary and pairwise feedback, proposing algorithms with theoretical guarantees and demonstrating query savings in experiments.

Contribution

It introduces novel algorithms for ensemble selection under different feedback types, with theoretical analysis and practical experiments showing improved efficiency.

Findings

Failure-conditioned greedy algorithm achieves $(1-1/e)$ guarantee with query savings.

A submodular relaxation supports efficient pairwise feedback-based selection.

Experiments demonstrate practical query savings and the importance of complementarity.

Abstract

Organizations increasingly deploy multiple AI systems across task domains, but selecting a small, high-performing ensemble can require costly model calls, benchmark runs, and human evaluation. We study this selection problem as a distributional variant of multiwinner voting: tasks are drawn from an unknown domain distribution, each task induces feedback over candidate experts, and a committee's value on a task is determined by its best-performing member. We analyze both binary feedback, for tasks with correct/incorrect outcomes, and pairwise feedback, for tasks where candidate outputs are compared by preference. In the binary setting, the induced objective is coverage. We give exhaustive-elicitation baselines and matching worst-case query lower bounds, and we design a failure-conditioned greedy algorithm that preserves the standard $(1-1/e)$ guarantee while obtaining instance-dependent query savings. In the pairwise setting, we study $\theta$-winning committees. We show that full-information optimization admits a PTAS but no EPTAS under Gap-ETH, and that the objective is monotone but not submodular. This motivates a weighted ordinal coverage relaxation, which is submodular and supports a failure-conditioned greedy oracle under pairwise feedback. We then convert this oracle back into $\theta$-type guarantees through finite-family auditing or a minimax wrapper. We also provide small-scale LLM experiments illustrating the predicted query savings and the role of complementarity in committee selection.