EvoRoute: Experience-Driven Self-Routing LLM Agent Systems

TL;DR

EvoRoute introduces an adaptive routing system for LLM-based agents that dynamically balances performance, cost, and speed by learning from experience, significantly reducing expenses and latency while maintaining high task effectiveness.

Contribution

It presents EvoRoute, a novel self-evolving model routing approach that optimizes LLM selection in real-time based on environment feedback, surpassing static routing methods.

Findings

Reduces execution cost by up to 80%

Lowers latency by over 70%

Maintains or improves system performance

Abstract

Complex agentic AI systems, powered by a coordinated ensemble of Large Language Models (LLMs), tool and memory modules, have demonstrated remarkable capabilities on intricate, multi-turn tasks. However, this success is shadowed by prohibitive economic costs and severe latency, exposing a critical, yet underexplored, trade-off. We formalize this challenge as the \textbf{Agent System Trilemma}: the inherent tension among achieving state-of-the-art performance, minimizing monetary cost, and ensuring rapid task completion. To dismantle this trilemma, we introduce EvoRoute, a self-evolving model routing paradigm that transcends static, pre-defined model assignments. Leveraging an ever-expanding knowledge base of prior experience, EvoRoute dynamically selects Pareto-optimal LLM backbones at each step, balancing accuracy, efficiency, and resource use, while continually refining its own selection policy through environment feedback. Experiments on challenging agentic benchmarks such as GAIA and BrowseComp+ demonstrate that EvoRoute, when integrated into off-the-shelf agentic systems, not only sustains or enhances system performance but also reduces execution cost by up to $80\%$ and latency by over $70\%$.