Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing

TL;DR

This paper presents an adaptive, multi-objective routing algorithm using NSGA-II to efficiently distribute LLM inference requests across heterogeneous cloud-edge environments, balancing quality, latency, and cost.

Contribution

It introduces a novel multi-objective optimization approach for LLM request routing that adapts to workload heterogeneity and resource diversity, improving scalability and efficiency.

Findings

Preserves 95.2% of cloud-only response quality

Reduces inference cost by 34.9%

Maintains slight latency increase

Abstract

The rising demand for Large Language Model (LLM) inference services has intensified pressure on computational resources, resulting in latency and cost challenges. This paper introduces a novel routing algorithm based on the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to distribute inference requests across heterogeneous LLM instances in a cloud-edge computing environment. Formulated as a multi-objective optimization problem, the algorithm balances response quality, response time, and inference cost, adapting to request heterogeneity (e.g., varying complexity and prompt lengths) and node diversity (e.g., edge vs. cloud resources). This adaptive routing algorithm optimizes performance under dynamic workloads. We benchmark the approach using a testbed with datasets including Stanford Question Answering Dataset (SQuAD), Mostly Basic Python Problems (MBPP), Hella Situations With Adversarial Generations (HellaSwag), and Grade School Math 8K (GSM8K). Experimental results show our solution, compared to the baselines, preserves 95.2% of Cloud-Only response quality with slight latency increase, while reducing inference cost by 34.9%. These findings validate the algorithm's effectiveness for scalable LLM deployments.