Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives

TL;DR

This paper advocates for using asymptotic analysis with LLM primitives to evaluate and improve the efficiency of decomposing complex problems into multiple LLM-based agents, facilitating scalable deployment.

Contribution

It introduces the concept of asymptotic analysis with LLM primitives as a tool to assess and optimize the decomposition of problems into LLM agents for better scalability.

Findings

Asymptotic analysis separates LLM computational cost from orchestration efficiency.

Decomposition strategies can be optimized using insights from asymptotic analysis.

Scaling LLM systems benefits from principled, non-anthropomorphic analysis methods.

Abstract

Decomposing hard problems into subproblems often makes them easier and more efficient to solve. With large language models (LLMs) crossing critical reliability thresholds for a growing slate of capabilities, there is an increasing effort to decompose systems into sets of LLM-based agents, each of whom can be delegated sub-tasks. However, this decomposition (even when automated) is often intuitive, e.g., based on how a human might assign roles to members of a human team. How close are these role decompositions to optimal? This position paper argues that asymptotic analysis with LLM primitives is needed to reason about the efficiency of such decomposed systems, and that insights from such analysis will unlock opportunities for scaling them. By treating the LLM forward pass as the atomic unit of computational cost, one can separate out the (often opaque) inner workings of a particular LLM from the inherent efficiency of how a set of LLMs are orchestrated to solve hard problems. In other words, if we want to scale the deployment of LLMs to the limit, instead of anthropomorphizing LLMs, asymptotic analysis with LLM primitives should be used to reason about and develop more powerful decompositions of large problems into LLM agents.