Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM

TL;DR

Cambricon-LLM introduces a novel chiplet-based hybrid architecture combining NPU and NAND flash to enable efficient on-device inference of large 70B LLMs on edge devices, overcoming memory and bandwidth limitations.

Contribution

This work presents a new hybrid hardware architecture with hardware-tiling, in-flash computing, and on-die ECC techniques for efficient large language model inference on edge devices.

Findings

Achieves 3.44 token/s for 70B LLMs on edge devices.

Over 22X to 45X speedup over existing flash-offloading methods.

Demonstrates feasibility of deploying large LLMs on resource-constrained devices.

Abstract

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses the significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash's on-die processing capabilities. Overall, Cambricon-LLM enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, and 7B LLMs at a speed of 36.34 token/s, which is over 22X to 45X faster than existing flash-offloading technologies, showing the potentiality of deploying powerful LLMs in edge devices.