Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection

TL;DR

This paper introduces Routing with Generated Data (RGD), a method for LLM model selection using only generated queries and answers, and proposes CASCAL, a robust query-only router that outperforms existing methods especially with lower-quality generators.

Contribution

The paper presents RGD for training routers without ground-truth data and introduces CASCAL, a new query-only routing approach that is more robust to generator quality variations.

Findings

Query-answer routers degrade faster with lower generator quality.

Effective generators must produce diverse and accurate responses.

CASCAL outperforms existing routers by 4.6% accuracy with weak generator data.

Abstract

Large Language Model (LLM) routers dynamically select optimal models for given inputs. Existing approaches typically assume access to ground-truth labeled data, which is often unavailable in practice, especially when user request distributions are heterogeneous and unknown. We introduce Routing with Generated Data (RGD), a challenging setting in which routers are trained exclusively on generated queries and answers produced from high-level task descriptions by generator LLMs. We evaluate query-answer routers (using both queries and labels) and query-only routers across four diverse benchmarks and 12 models, finding that query-answer routers degrade faster than query-only routers as generator quality decreases. Our analysis reveals two crucial characteristics of effective generators: they must accurately respond to their own questions, and their questions must produce sufficient performance differentiation among the model pool. We then show how filtering for these characteristics can improve the quality of generated data. We further propose CASCAL, a novel query-only router that estimates model correctness through consensus voting and identifies model-specific skill niches via hierarchical clustering. CASCAL is substantially more robust to generator quality, outperforming the best query-answer router by 4.6% absolute accuracy when trained on weak generator data.