Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers

TL;DR

This paper demonstrates that a simple kNN approach can outperform complex learned routers for LLM model selection across various tasks, emphasizing the value of simplicity and standardized benchmarks.

Contribution

It introduces a suite of routing benchmarks and shows that kNN-based routing often surpasses state-of-the-art learned methods, challenging the trend towards complex architectures.

Findings

kNN matches or outperforms learned routers across tasks

Locality in embedding space enables effective simple routing

Standardized benchmarks facilitate fair comparison

Abstract

As large language models (LLMs) grow in scale and specialization, routing--selecting the best model for a given input--has become essential for efficient and effective deployment. While recent methods rely on complex learned routing strategies, their dependence on disparate training data and evaluation setups makes comparison and generalization difficult. In this work, we revisit LLM routing through the lens of simplicity. We show that a well-tuned k-Nearest Neighbors (kNN) approach not only matches but often outperforms state-of-the-art learned routers across diverse tasks. To support systematic evaluation, we introduce a suite of standardized routing benchmarks spanning instruction-following, question-answering, and reasoning tasks, as well as the first multi-modal routing dataset involving visual inputs. Our findings reveal that the locality properties of model performance in embedding space enable simple non-parametric methods to achieve strong routing decisions with lower sample complexity than parametric approaches. This challenges the prevailing trend toward sophisticated architectures and highlights the importance of thoroughly evaluating simple baselines before investing in complex solutions. To support reproducibility and further exploration, we will release all benchmarks and code upon publication.