Asymptotically Optimal Sequential Testing with Heterogeneous LLMs

TL;DR

This paper develops an asymptotically optimal Bayesian sequential testing strategy for multiple heterogeneous large language models, balancing query and waiting costs, and demonstrating that at most two models are needed for optimality as error tolerance diminishes.

Contribution

It introduces belief-dependent policies that optimally combine two LLMs for hypothesis testing, extending previous single-model approaches and proving asymptotic optimality.

Findings

Optimal policies use at most two LLMs asymptotically.

Single-LLM policies are not generally optimal under asymmetry.

Constructed policies match the lower bound up to a small factor as error decreases.

Abstract

We study a Bayesian binary sequential hypothesis testing problem with multiple large language models (LLMs). Each LLM $j$ has per-query cost $c_j>0$, random waiting time with mean $\mu_j>0$ and sub-Gaussian tails, and \emph{asymmetric} accuracies: the probability of returning the correct label depends on the true hypothesis $\theta\in\{A,B\}$ and needs not be the same under $A$ and $B$. This asymmetry induces two distinct information rates $(I_{j,A}, I_{j,B})$ per LLM, one under each hypothesis. The decision-maker chooses LLMs sequentially, observes their noisy binary answers, and stops when the posterior probability of one hypothesis exceeds $1-\alpha$. The objective is to minimize the sum of expected query cost and expected waiting cost, $\mathbb{E}[C_\pi] + \mathbb{E}[g(W_\pi)]$, where $C_\pi$ is the total query cost, $W_\pi$ is the total waiting time and $g$ is a polynomial function (e.g., $g(x)=x^\rho$ with $\rho\ge 1$). We prove that as the error tolerance $\alpha\to0$, the optimal policy is asymptotically equivalent to one that uses at most two LLMs. In this case, a single-LLM policy is \emph{not} generically optimal: optimality now requires exploiting a two-dimensional tradeoff between information under $A$ and information under $B$. Any admissible policy induces an expected information-allocation vector in $\mathbb{R}_+^2$, and we show that the optimal allocation lies at an extreme point of the associated convex set when $\alpha$ is relatively small, and hence uses at most two LLMs. We construct belief-dependent policies that first mix between two LLMs when the posterior is ambiguous, and then switch to a single "specialist" LLM when the posterior is sufficiently close to one of the hypotheses. These policies match the universal lower bound up to a $(1+o(1))$ factor as $\alpha\rightarrow 0$.