Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space

TL;DR

ZeroRouter introduces a universal latent space for LLM routing, enabling zero-shot model onboarding and significantly reducing retraining costs while improving accuracy, cost-efficiency, and latency in model selection.

Contribution

It proposes a universal latent space for LLM routing that decouples query characterization from model profiling, allowing zero-shot onboarding of new models without retraining.

Findings

Outperforms baselines in accuracy, cost, and latency

Enables zero-shot onboarding of new models

Reduces retraining costs significantly

Abstract

The rapid proliferation of Large Language Models (LLMs) has led to a fragmented and inefficient ecosystem, a state of ``model lock-in'' where seamlessly integrating novel models remains a significant bottleneck. Current routing frameworks require exhaustive, costly retraining, hindering scalability and adaptability. We introduce ZeroRouter, a new paradigm for LLM routing that breaks this lock-in. Our approach is founded on a universal latent space, a model-agnostic representation of query difficulty that fundamentally decouples the characterization of a query from the profiling of a model. This allows for zero-shot onboarding of new models without full-scale retraining. ZeroRouter features a context-aware predictor that maps queries to this universal space and a dual-mode optimizer that balances accuracy, cost, and latency. Our framework consistently outperforms all baselines, delivering higher accuracy at lower cost and latency.