UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
Varun Kotte

TL;DR
UCCI is a calibration-first routing method for LLM cascades that minimizes inference costs by accurately mapping uncertainty to error probability, outperforming existing methods on real workloads.
Contribution
The paper introduces UCCI, a novel calibration-based routing approach that optimally selects escalation thresholds using isotonic regression, reducing inference costs in LLM cascades.
Findings
UCCI reduces inference cost by 31% on a production NER workload.
UCCI achieves lower calibration error (ECE 0.03) compared to uncalibrated methods.
UCCI outperforms entropy thresholding, split-conformal routing, and learned thresholds.
Abstract
LLM cascades and model routing promise lower inference cost by sending easy queries to a small model and escalating hard ones to a large model, but most deployed routers use uncalibrated confidence scores and require per-workload threshold tuning. We present UCCI, a calibration-first router that maps token-level margin uncertainty to a per-query error probability via isotonic regression and selects the escalation threshold by constrained cost minimization. Under three explicit assumptions, threshold policies on the calibrated score are cost-optimal, and isotonic calibration achieves O(n^{-1/3}) sample complexity for expected calibration error (ECE). On a production named entity recognition workload of 75,000 queries served by 4B and 12B instruction-tuned LLMs on H100 GPUs, UCCI cuts inference cost by 31% (95% CI: [27%, 35%]) at micro-F1 = 0.91 while reducing ECE from 0.12 to 0.03. At…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
