RouterBench: A Benchmark for Multi-LLM Routing System
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin,, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay

TL;DR
RouterBench introduces a standardized evaluation framework and dataset for assessing multi-LLM routing systems, facilitating improved development and comparison of strategies for efficient and cost-effective LLM deployment.
Contribution
The paper presents RouterBench, a novel benchmark and dataset for evaluating LLM routing systems, along with a theoretical framework and comparative analysis of routing approaches.
Findings
RouterBench provides over 405k inference outcomes for evaluation.
Comparative analysis highlights strengths and limitations of different routing strategies.
The framework advances standardized assessment of LLM routing systems.
Abstract
As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. Yet, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To bridge this gap, we present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies. We further…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIPv6, Mobility, Handover, Networks, Security · Cooperative Communication and Network Coding · Mobile Agent-Based Network Management
