RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization

TL;DR

RouteNLP is a cost-effective, closed-loop framework that intelligently routes NLP queries across models, reducing inference costs by up to 58% while maintaining high quality and low latency in real-world and benchmark settings.

Contribution

The paper introduces a novel routing framework combining confidence calibration, conformal prediction, and distillation co-optimization to significantly reduce NLP inference costs.

Findings

Reduced inference costs by 58% in real deployment.

Achieved 40-85% cost reduction on benchmark tasks.

Maintained high quality with 96-100% accuracy and 74.5% human-rated output quality.

Abstract

Serving diverse NLP workloads with large language models is costly: at one enterprise partner, inference costs exceeded $200K/month despite over 70% of queries being routine tasks well within the capability of smaller models. We present RouteNLP, a closed-loop framework that routes queries across a tiered model portfolio to minimize cost while satisfying per-task quality constraints. The framework integrates three components: a difficulty-aware router with shared task-conditioned representations trained on preference data and quality signals; confidence-calibrated cascading that uses conformal prediction for distribution-free threshold initialization; and a distillation-routing co-optimization loop that clusters escalation failures, applies targeted knowledge distillation to cheaper models, and automatically retrains the router, yielding over twice the cost improvement of untargeted distillation. In an 8-week pilot deployment processing ~5K queries/day at an enterprise customer-service division, RouteNLP reduced inference costs by 58% while maintaining 91% response acceptance and reducing p99 latency from 1,847 ms to 387 ms. On a six-task benchmark spanning finance, customer service, and legal domains, the framework achieves 40-85% cost reduction while retaining 96-100% quality on structured tasks and 96-98% on generation tasks, with human evaluation confirming that 74.5% of routed generation outputs match or exceed frontier-model quality.