Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations

TL;DR

This paper introduces a federated learning framework for language model routing that enables multiple clients to collaboratively learn effective routing policies without sharing sensitive data, improving model coverage and cost-efficiency.

Contribution

It presents the first federated approach for LLM routing, supporting both parametric and nonparametric routers under diverse client data distributions.

Findings

Federated training improves routing accuracy and cost-efficiency.

The framework enhances model coverage and query generalization.

Theoretical analysis confirms reduced routing suboptimality.

Abstract

Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models locally. Since models vary widely in capability and price, routing queries to models that balance quality and inference cost is essential. Existing router approaches assume access to centralized query-model evaluation data. However, these data are often fragmented across clients, such as end users and organizations, and are privacy-sensitive, which makes centralizing data infeasible. Additionally, per-client router training is ineffective since local evaluation data is limited and covers only a restricted query distribution and a biased subset of model evaluations. We introduce the first federated framework for LLM routing, enabling clients to learn a shared routing policy from local offline query-model evaluation data. Our framework supports both parametric multilayer perceptron router and nonparametric K-means router under heterogeneous client query distributions and non-uniform model coverage. Across two benchmarks, federated collaboration improves the accuracy-cost frontier over client-local routers, both via increased effective model coverage and better query generalization. Our theoretical results also validate that federated training reduces routing suboptimality.