Accuracy-Delay Trade-Off in LLM Offloading via Token-Level Uncertainty

TL;DR

This paper introduces a token-level uncertainty-aware offloading framework for LLMs in mobile edge computing, balancing inference accuracy and latency by dynamically deciding between local computation and offloading based on uncertainty metrics.

Contribution

It proposes a novel margin-based token-level uncertainty metric and a greedy offloading algorithm that improves delay-accuracy trade-offs in MEC environments.

Findings

GOA outperforms baseline strategies in accuracy and latency.

The uncertainty metric correlates well with model accuracy.

The framework is scalable and practical for real-world MEC settings.

Abstract

Large language models (LLMs) offer significant potential for intelligent mobile services but are computationally intensive for resource-constrained devices. Mobile edge computing (MEC) allows such devices to offload inference tasks to edge servers (ESs), yet introduces latency due to communication and serverside queuing, especially in multi-user environments. In this work, we propose an uncertainty-aware offloading framework that dynamically decides whether to perform inference locally or offload it to the ES, based on token-level uncertainty and resource constraints. We define a margin-based token-level uncertainty metric and demonstrate its correlation with model accuracy. Leveraging this metric, we design a greedy offloading algorithm (GOA) that minimizes delay while maintaining accuracy by prioritizing offloading for highuncertainty queries. Our experiments show that GOA consistently achieves a favorable trade-off, outperforming baseline strategies in both accuracy and latency across varying user densities, and operates with practical computation time. These results establish GOA as a scalable and effective solution for LLM inference in MEC environments.