Investigating Energy Bounds of Analog Compute-in-Memory with Local Normalization

TL;DR

This paper introduces local normalization in analog compute-in-memory architectures, enabling increased dynamic range and energy efficiency for AI workloads, especially large language models, by using a gain-ranging MAC.

Contribution

It proposes a novel gain-ranging MAC with local normalization, reducing ADC resolution dependence and improving energy efficiency in analog CIM for AI applications.

Findings

Enables 4-bit increase in input dynamic range without extra energy

ADC resolution becomes input distribution invariant

Establishes an energy-efficient upper bound for analog CIM

Abstract

Modern edge AI workloads demand maximum energy efficiency, motivating the pursuit of analog Compute-in-Memory (CIM) architectures. Simultaneously, the popularity of Large-Language-Models (LLMs) drives the adoption of low-bit floating-point formats which prioritize dynamic range. However, the conventional direct-accumulation CIM accommodates floating-points by normalizing them to a shared widened fixed-point scale. Consequently, hardware resolution is dictated by the input's dynamic range rather than its precision, and energy consumption is dominated by the ADC. We address this limitation by introducing local normalization for each input, weight, and multiply-accumulate (MAC) output via a Gain-Ranging MAC (GR-MAC). Normalization overhead is handled by low-power digital logic, enabling the computationally expensive MAC operation to remain in the energy-efficient low-precision analog regime. Energy modelling shows that the addition of a gain-ranging Stage to the MAC enables a 4-bit increase in input dynamic range without increased energy consumption at a 35 dB SQNR standard. Additionally, the ADC resolution requirement becomes invariant to input distribution assumptions, allowing construction of an upper bound with a 1.5-bit reduction compared to the conventional lower bound. These results establish a pathway towards unlocking favourable energy scaling trends of analog CIM for modern AI workloads.