RCW-CIM: A Digital CIM-based LLM Accelerator with Read-Compute/Write

TL;DR

This paper introduces RCW-CIM, a novel digital CIM-based LLM accelerator that minimizes weight update latency and improves overall performance through innovative architecture and dataflow optimizations.

Contribution

It proposes a read-compute/write architecture with nonlinear operator fusion and a new dataflow, significantly reducing latency and DRAM access for LLM acceleration.

Findings

Decoding latency reduced by 21.59% on Llama2-7B.

Latency reduced by 69.17% through nonlinear operator fusion.

DRAM access and CIM weight updates reduced by 51.6% and 87.6%.

Abstract

Digital computing-in-memory (DCIM) has emerged as a promising solution for large language model (LLM) acceleration by minimizing data transfers between external DRAM and on-chip accelerators while maintaining high precision for superior accuracy. However, existing CIM architectures often overlook weight update latency, which becomes critical as LLM weights are far larger than a single CIM macro capacity. To address this issue, this paper proposes a read-compute/write (RCW) architecture that effectively minimizes weight update latency, along with a nonlinear operator fusion that further mitigates dependencyinduced latency. The proposed RCW reduces decoding computing latency by 21.59% on the Llama2-7B model. In addition, the nonlinear operator fusion mechanism achieves a 69.17% latency reduction through efficient partial accumulation and group-based approximation. Furthermore, a weight-stationary and output column stationary (WS-OCS) dataflow is introduced to reduce both external DRAM access and internal CIM weight updates by 51.6% and 87.6% respectively during the prefill phase of 1024 tokens, leading to an overall 49.76% latency reduction. Fabricated using TSMC 22 nm CMOS technology and operating at 100 MHz, the proposed RCW-CIM achieves 3.28 TOPS and 42.3 TOPS/W, enabling 4.2 ms prefill latency and 26.87 decoded tokens per second for the INT4-weight Llama2 model with dual DDR5-6400 memory.