BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

TL;DR

BEST-Route is a novel framework for adaptive LLM query routing that dynamically selects models and response counts based on query difficulty, significantly reducing costs while maintaining high performance.

Contribution

It introduces a new routing approach that optimally chooses the number of responses from small models to improve quality-cost trade-offs in LLM deployment.

Findings

Reduces LLM deployment costs by up to 60%.

Maintains less than 1% performance drop.

Effectively balances model quality and response sampling.

Abstract

Large language models (LLMs) are powerful tools but are often expensive to deploy at scale. LLM query routing mitigates this by dynamically assigning queries to models of varying cost and quality to obtain a desired trade-off. Prior query routing approaches generate only one response from the selected model and a single response from a small (inexpensive) model was often not good enough to beat a response from a large (expensive) model due to which they end up overusing the large model and missing out on potential cost savings. However, it is well known that for small models, generating multiple responses and selecting the best can enhance quality while remaining cheaper than a single large-model response. We leverage this idea to propose BEST-Route, a novel routing framework that chooses a model and the number of responses to sample from it based on query difficulty and the quality thresholds. Experiments on real-world datasets demonstrate that our method reduces costs by up to 60% with less than 1% performance drop.