Recursive Offloading for LLM Serving in Multi-tier Networks

TL;DR

RecServe is a recursive offloading framework that optimizes LLM inference in multi-tier networks by dynamically adjusting offloading decisions based on task complexity and confidence, significantly reducing communication overhead and improving service quality.

Contribution

The paper introduces RecServe, a novel recursive offloading framework with hierarchical confidence evaluation and dynamic thresholding for efficient LLM serving across device, edge, and cloud tiers.

Findings

RecServe reduces communication overhead by over 50% compared to centralized cloud serving.

RecServe outperforms existing methods like CasServe in service quality.

Experimental results on eight datasets validate RecServe's effectiveness.

Abstract

Heterogeneous device-edge-cloud computing infrastructures have become widely adopted in telecommunication operators and Wide Area Networks (WANs), offering multi-tier computational support for emerging intelligent services. With the rapid proliferation of Large Language Model (LLM) services, efficiently coordinating inference tasks and reducing communication overhead within these multi-tier network architectures becomes a critical deployment challenge. Existing LLM serving paradigms exhibit significant limitations: on-device deployment supports only lightweight LLMs due to hardware constraints, while cloud-centric deployment suffers from resource congestion and considerable prompt communication overhead caused by frequent service requests during peak periods. Although the model-cascading-based inference strategy adapts better to multi-tier networks, its reliance on fine-grained, manually adjusted thresholds makes it less responsive to dynamic network conditions and varying task complexities. To address these challenges, we propose RecServe, a recursive offloading framework tailored for LLM serving in multi-tier networks. RecServe integrates a task-specific hierarchical confidence evaluation mechanism that guides offloading decisions based on inferred task complexity in progressively scaled LLMs across device, edge, and cloud tiers. To further enable intelligent task routing across tiers, RecServe employs a sliding-window-based dynamic offloading strategy with quantile interpolation, enabling real-time tracking of historical confidence distributions and adaptive offloading threshold adjustments. Experiments on eight datasets demonstrate that RecServe outperforms CasServe in both service quality and communication efficiency, and reduces the communication burden by over 50\% compared to centralized cloud-based serving.