Scalable Prompt Routing via Fine-Grained Latent Task Discovery

TL;DR

This paper introduces a scalable prompt routing system that automatically discovers fine-grained task types and uses a two-stage architecture to improve model selection, achieving better performance and cost-efficiency across multiple benchmarks.

Contribution

It presents a novel two-stage routing architecture with automated task discovery and task-aware quality estimation, addressing limitations of manual taxonomies and monolithic routers.

Findings

Outperforms existing baselines on 10 benchmarks

Surpasses the strongest individual model in performance

Reduces cost to less than half of the strongest model

Abstract

Prompt routing dynamically selects the most appropriate large language model from a pool of candidates for each query, optimizing performance while managing costs. As model pools scale to include dozens of frontier models with narrow performance gaps, existing approaches face significant challenges: manually defined task taxonomies cannot capture fine-grained capability distinctions, while monolithic routers struggle to differentiate subtle differences across diverse tasks. We propose a two-stage routing architecture that addresses these limitations through automated fine-grained task discovery and task-aware quality estimation. Our first stage employs graph-based clustering to discover latent task types and trains a classifier to assign prompts to discovered tasks. The second stage uses a mixture-of-experts architecture with task-specific prediction heads for specialized quality estimates. At inference, we aggregate predictions from both stages to balance task-level stability with prompt-specific adaptability. Evaluated on 10 benchmarks with 11 frontier models, our method consistently outperforms existing baselines and surpasses the strongest individual model while incurring less than half its cost.