Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference

TL;DR

This paper introduces HeteroInfer, a novel mobile LLM inference engine that efficiently utilizes both GPU and NPU accelerators, achieving significant speedups by exploiting heterogeneous processing and memory bandwidth.

Contribution

It presents a comprehensive characterization of mobile SoC heterogeneity and proposes mechanisms for joint GPU-NPU utilization, improving LLM inference performance on mobile devices.

Findings

HeteroInfer achieves up to 6.02x speedup over existing engines.

The proposed synchronization mechanism reduces overhead.

Efficient resource utilization with negligible impact on other apps.

Abstract

With the rapid advancement of artificial intelligence technologies such as ChatGPT, AI agents, and video generation, contemporary mobile systems have begun integrating these AI capabilities on local devices to enhance privacy and reduce response latency. To meet the computational demands of AI tasks, current mobile SoCs are equipped with diverse AI accelerators, including GPUs and Neural Processing Units (NPUs). However, there has not been a comprehensive characterization of these heterogeneous processors, and existing designs typically only leverage a single AI accelerator for LLM inference, leading to suboptimal use of computational resources and memory bandwidth. In this paper, we first summarize key performance characteristics of heterogeneous processors, SoC memory bandwidth, etc. Drawing on these observations, we propose different heterogeneous parallel mechanisms to fully exploit both GPU and NPU computational power and memory bandwidth. We further design a fast synchronization mechanism between heterogeneous processors that leverages the unified memory architecture. By employing these techniques, we present HeteroInfer, the fastest LLM inference engine in mobile devices which supports GPU-NPU heterogeneous execution. Evaluation shows that HeteroInfer delivers a 1.34x to 6.02x end-to-end speedup over state-of-the-art GPU-only and NPU-only LLM engines, while maintaining negligible interference with other applications.