Universal Model Routing for Efficient LLM Inference

TL;DR

UniRoute introduces a dynamic model routing method for large language models that efficiently handles unseen models at test time by representing LLMs as feature vectors and using clustering techniques.

Contribution

The paper presents UniRoute, a novel approach for dynamic LLM routing that generalizes to unseen models using feature-based representations and clustering, with theoretical guarantees.

Findings

Effective routing among 30+ unseen LLMs

Outperforms fixed pool routing methods

Theoretically grounded with excess risk bounds

Abstract

Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective instantiations of UniRoute, relying on cluster-based routing and a learned cluster map respectively. We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound. Experiments on a range of public benchmarks show the effectiveness of UniRoute in routing amongst more than 30 unseen LLMs.