Leveraging Uncertainty Estimation for Efficient LLM Routing

TL;DR

This paper introduces a Confidence-Driven LLM Router that uses uncertainty estimation to improve routing decisions, balancing cost and response quality by simulating human preferences and outperforming existing methods.

Contribution

It presents a novel framework leveraging uncertainty estimation for LLM routing, incorporating human-like response quality assessment, and demonstrating superior performance over existing methods.

Findings

Outperforms state-of-the-art routing methods in experiments

Achieves better response quality while maintaining cost efficiency

Introduces LLM-as-a-Judge for response quality assessment

Abstract

Deploying large language models (LLMs) in edge-cloud environments requires an efficient routing strategy to balance cost and response quality. Traditional approaches prioritize either human-preference data or accuracy metrics from benchmark datasets as routing criteria, but these methods suffer from rigidity and subjectivity. Moreover, existing routing frameworks primarily focus on accuracy and cost, neglecting response quality from a human preference perspective. In this work, we propose the Confidence-Driven LLM Router, a novel framework that leverages uncertainty estimation to optimize routing decisions. To comprehensively assess routing performance, we evaluate both system cost efficiency and response quality. In particular, we introduce the novel use of LLM-as-a-Judge to simulate human rating preferences, providing the first systematic assessment of response quality across different routing strategies. Extensive experiments on MT-Bench, GSM8K, and MMLU demonstrate that our approach outperforms state-of-the-art routing methods, achieving superior response quality while maintaining cost efficiency.