Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization

TL;DR

This paper proposes an optimization framework for deploying large language models on edge devices using batching and quantization, improving inference throughput while considering resource constraints and user requirements.

Contribution

It introduces a novel edge inference optimization problem for LLMs, with a new algorithm (DFTSP) that enhances throughput and reduces complexity compared to existing methods.

Findings

DFTSP outperforms batching benchmarks in throughput.

DFTSP reduces time complexity by over 45%.

Simulation shows improved resource utilization and latency management.

Abstract

Generative Artificial Intelligence (GAI) is taking the world by storm with its unparalleled content creation ability. Large Language Models (LLMs) are at the forefront of this movement. However, the significant resource demands of LLMs often require cloud hosting, which raises issues regarding privacy, latency, and usage limitations. Although edge intelligence has long been utilized to solve these challenges by enabling real-time AI computation on ubiquitous edge resources close to data sources, most research has focused on traditional AI models and has left a gap in addressing the unique characteristics of LLM inference, such as considerable model size, auto-regressive processes, and self-attention mechanisms. In this paper, we present an edge intelligence optimization problem tailored for LLM inference. Specifically, with the deployment of the batching technique and model quantization on resource-limited edge devices, we formulate an inference model for transformer decoder-based LLMs. Furthermore, our approach aims to maximize the inference throughput via batch scheduling and joint allocation of communication and computation resources, while also considering edge resource constraints and varying user requirements of latency and accuracy. To address this NP-hard problem, we develop an optimal Depth-First Tree-Searching algorithm with online tree-Pruning (DFTSP) that operates within a feasible time complexity. Simulation results indicate that DFTSP surpasses other batching benchmarks in throughput across diverse user settings and quantization techniques, and it reduces time complexity by over 45% compared to the brute-force searching method.