Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism

TL;DR

Ghidorah is a system that accelerates large language model inference on end-user devices by using speculative decoding and heterogeneous processing units, achieving significant speedups while optimizing resource utilization.

Contribution

The paper introduces Ghidorah, a novel inference system combining speculative decoding with hetero-core parallelism tailored for unified memory architectures in end-user devices.

Findings

Achieves up to 7.6x speedup in LLM decoding phase

Effectively utilizes heterogeneous processing units

Optimizes sparse computation on ARM CPUs

Abstract

In-situ LLM inference on end-user devices has gained significant interest due to its privacy benefits and reduced dependency on external infrastructure. However, as the decoding process is memory-bandwidth-bound, the diverse processing units in modern end-user devices cannot be fully exploited, resulting in slow LLM inference. This paper presents Ghidorah, a LLM inference system for end-user devices with the unified memory architecture. The key idea of Ghidorah can be summarized in two steps: 1) leveraging speculative decoding approaches to enhance parallelism, and 2) ingeniously distributing workloads across multiple heterogeneous processing units to maximize computing power utilization. Ghidorah includes the hetero-core model parallelism (HCMP) architecture and the architecture-aware profiling (ARCA) approach. The HCMP architecture guides partitioning by leveraging the unified memory design of end-user devices and adapting to the hybrid computational demands of speculative decoding. The ARCA approach is used to determine the optimal speculative strategy and partitioning strategy, balancing acceptance rate with parallel capability to maximize the speedup. Additionally, we optimize sparse computation on ARM CPUs. Experimental results show that Ghidorah can achieve up to 7.6x speedup in the dominant LLM decoding phase compared to the sequential decoding approach in NVIDIA Jetson NX.