HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network

TL;DR

HALO is a framework that enhances distributed large language model inference at the edge by using semantic-aware synchronization and load balancing, achieving significant speedups despite lossy network conditions.

Contribution

HALO introduces a semantic-aware predictor, parallel neuron loading, and load balancing to improve distributed LLM inference in unreliable edge networks, reducing synchronization delays.

Findings

3.41x end-to-end speedup on Raspberry Pi cluster

Maintains performance comparable to ideal conditions

Outperforms existing methods in lossy network scenarios

Abstract

The deployment of large language models' (LLMs) inference at the edge can facilitate prompt service responsiveness while protecting user privacy. However, it is critically challenged by the resource constraints of a single edge node. Distributed inference has emerged to aggregate and leverage computational resources across multiple devices. Yet, existing methods typically require strict synchronization, which is often infeasible due to the unreliable network conditions. In this paper, we propose HALO, a novel framework that can boost the distributed LLM inference in lossy edge network. The core idea is to enable a relaxed yet effective synchronization by strategically allocating less critical neuron groups to unstable devices, thus avoiding the excessive waiting time incurred by delayed packets. HALO introduces three key mechanisms: (1) a semantic-aware predictor to assess the significance of neuron groups prior to activation. (2) a parallel execution scheme of neuron group loading during the model inference. (3) a load-balancing scheduler that efficiently orchestrates multiple devices with heterogeneous resources. Experimental results from a Raspberry Pi cluster demonstrate that HALO achieves a 3.41x end-to-end speedup for LLaMA-series LLMs under unreliable network conditions. It maintains performance comparable to optimal conditions and significantly outperforms the state-of-the-art in various scenarios.