SHIELD: A Segmented Hierarchical Memory Architecture for Energy-Efficient LLM Inference on Edge NPUs

TL;DR

SHIELD is a novel segmented eDRAM architecture that reduces energy consumption for LLM inference on edge NPUs by exploiting activation properties and lifecycle awareness.

Contribution

It introduces a lifecycle-aware segmented eDRAM design that selectively disables or relaxes refresh based on activation transientness and sensitivity.

Findings

Reduces eDRAM refresh energy by 35%

Maintains accuracy on WikiText-2, PIQA, and ARC-Easy datasets

Applicable across multiple LLMs and inference scenarios

Abstract

Large Language Model (LLM) inference on edge Neural Processing Units (NPUs) is fundamentally constrained by limited on-chip memory capacity. Although high-density embedded DRAM (eDRAM) is attractive for storing activation workspaces, its periodic refresh consumes substantial energy. Prior work has primarily focused on reducing off-chip traffic or optimizing refresh for persistent Key-Value (KV) caches, while transient and error-resilient Query and Attention Output (QO) activations are largely overlooked. We propose SHIELD, a lifecycle-aware segmented eDRAM architecture that jointly exploits temporal residency and bit-level sensitivity in bfloat16 (BF16) activations. SHIELD isolates the sign and exponent fields from the mantissa, disables refresh for transient QO mantissas, and applies relaxed refresh to persistent KV mantissas. Across multiple LLMs and inference scenarios, SHIELD reduces eDRAM refresh energy by 35% relative to a standard-refresh baseline while preserving accuracy on WikiText-2, PIQA, and ARC-Easy.