LLM-Driven Intent-Based Privacy-Aware Orchestration Across the Cloud-Edge Continuum

TL;DR

This paper presents a dynamic pipeline reconfiguration method for large language model inference in serverless cloud-edge environments, enabling online adjustments with minimal downtime and overhead.

Contribution

It introduces a novel online reconfiguration approach for LLM inference pipelines that adapts to workload changes on heterogeneous GPU clusters.

Findings

Service downtime is less than 50 ms during reconfiguration

Overhead on TTFT and TPOT is under 10%

Effective on NVIDIA A100 and L40s GPUs

Abstract

With the rapid advancement of large language models (LLMs), efficiently serving LLM inference under limited GPU resources has become a critical challenge. Recently, an increasing number of studies have explored applying serverless computing paradigms to LLM serving in order to maximize resource utilization. However, LLM inference workloads are highly diverse, and modern GPU clusters are inherently heterogeneous, making it necessary to dynamically adjust deployment configurations online to better adapt to the elastic and dynamic nature of serverless environments. At the same time, enabling such online reconfiguration is particularly challenging due to the stateful nature of LLM inference and the massive size of model parameters. In this paper, we propose a dynamic pipeline reconfiguration approach that enables online adjustment of pipeline configurations while minimizing service downtime and performance degradation. Our method allows the system to select the optimal pipeline configuration in response to changing workloads. Experimental results on heterogeneous GPU platforms, including NVIDIA A100 and L40s, demonstrate that our migration mechanism incurs less than 50 ms of service downtime, while introducing under 10% overhead on both time-to-first-token (TTFT) and time-per-output-token (TPOT).