AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality

TL;DR

AcceLLM introduces a redundancy-based approach to improve large language model inference by balancing load and optimizing data locality, significantly reducing latency and increasing efficiency in cloud GPU systems.

Contribution

It presents a novel method leveraging data redundancy to enhance load balancing and data locality for LLM inference, addressing latency and resource utilization issues.

Findings

Up to 30% reduction in latency and efficiency improvements over state-of-the-art systems.

Effective handling of diverse workloads with improved hardware utilization.

Validated on Nvidia H100 GPU and Huawei Ascend 910B2.

Abstract

Large Language Model (LLM) inference on large-scale systems is expected to dominate future cloud infrastructures. Efficient LLM inference in cloud environments with numerous AI accelerators is challenging, necessitating extensive optimizations for optimal performance. Current systems batch prefill and decoding to boost throughput but encounter latency issues, while others disaggregate these phases, leading to resource underutilization. We propose AcceLLM, a novel method addressing latency and load balancing, inspired by the cache data management. It strategically utilizes redundant data to enhance inference via load balancing and optimal hardware use. Simulated evaluations on Nvidia H100 GPU and Huawei Ascend 910B2 show AcceLLM surpasses state-of-the-art systems up to 30% in latency and efficiency, handling diverse workloads effectively.