Routing, Cascades, and User Choice for LLMs

TL;DR

This paper models the interaction between LLM providers and users, analyzing how routing and cascading decisions impact user utility and provider costs, revealing conditions for optimal policies and potential misalignments.

Contribution

It introduces a game-theoretic framework for LLM routing and user behavior, providing insights into optimal policies and misalignment issues.

Findings

Optimal routing often involves static policies without cascading.

Misalignment occurs when user and provider utility rankings differ.

Throttling can reduce costs but also depress user utility.

Abstract

To mitigate the trade-offs between performance and costs, LLM providers route user tasks to different models based on task difficulty and latency. We study the effect of LLM routing with respect to user behavior. We propose a game between an LLM provider with two models (standard and reasoning) and a user who can re-prompt or abandon tasks if the routed model cannot solve them. The user's goal is to maximize their utility minus the delay from using the model, while the provider minimizes the cost of servicing the user. We solve this Stackelberg game by fully characterizing the user best response and simplifying the provider problem. We observe that in nearly all cases, the optimal routing policy involves a static policy with no cascading that depends on the expected utility of the models to the user. Furthermore, we reveal a misalignment gap between the provider-optimal and user-preferred routes when the user's and provider's rankings of the models with respect to utility and cost differ. Finally, we demonstrate conditions for extreme misalignment where providers are incentivized to throttle the latency of the models to minimize their costs, consequently depressing user utility. The results yield simple threshold rules for single-provider, single-user interactions and clarify when routing, cascading, and throttling help or harm.