LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference

TL;DR

This paper presents LPU, a latency-optimized, scalable processor architecture designed to accelerate large language model inference, achieving significant speed and energy efficiency improvements over GPUs.

Contribution

The paper introduces LPU, a novel processor architecture with a streamlined dataflow and expandable synchronization, optimized for large language model inference.

Findings

LPU achieves 1.25 ms/token for 1.3B models, outperforming GPUs.

LPU is 2.09x faster than GPU for 1.3B models.

LPU-based servers are more energy-efficient than NVIDIA H100 and L4 servers.

Abstract

The explosive arrival of OpenAI's ChatGPT has fueled the globalization of large language model (LLM), which consists of billions of pretrained parameters that embodies the aspects of syntax and semantics. HyperAccel introduces latency processing unit (LPU), a latency-optimized and highly scalable processor architecture for the acceleration of LLM inference. LPU perfectly balances the memory bandwidth and compute logic with streamlined dataflow to maximize performance and efficiency. LPU is equipped with expandable synchronization link (ESL) that hides data synchronization latency between multiple LPUs. HyperDex complements LPU as an intuitive software framework to run LLM applications. LPU achieves 1.25 ms/token and 20.9 ms/token for 1.3B and 66B model, respectively, which is 2.09x and 1.37x faster than the GPU. LPU, synthesized using Samsung 4nm process, has total area of 0.824 mm2 and power consumption of 284.31 mW. LPU-based servers achieve 1.33x and 1.32x energy efficiency over NVIDIA H100 and L4 servers, respectively.