KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing

TL;DR

KVNAND introduces a novel DRAM-free in-flash computing architecture that stores both model weights and key-value caches in 3D NAND flash, enabling efficient large language model inference on resource-constrained edge devices.

Contribution

It presents the first DRAM-free IFC-based design for LLM inference, employing flash-based storage for weights and KV caches, and introduces a design space exploration framework for optimal configuration.

Findings

Achieves nearly 2x speedup over DRAM-based IFC at various context lengths.

Successfully handles 100K token context length without out-of-memory failures.

Mitigates latency, energy, and reliability issues associated with flash-based KV storage.

Abstract

Deploying large language models (LLMs) on edge devices enables personalized agents with strong privacy and low cost. However, with tens to hundreds of billions of parameters, single-batch autoregressive inference suffers from extremely low arithmetic intensity, creating severe weight-loading and bandwidth pressures on resource-constrained platforms. Recent in-flash computing (IFC) solutions alleviate this bottleneck by co-locating weight-related linear computations in the decode phase with flash, yet still rely on DRAM for the key-value (KV) cache. As context length grows, the KV cache can exceed model weights in size, imposing prohibitive DRAM cost and capacity requirements. Attempts to offload KV cache to flash suffer from severe performance penalties. We propose KVNAND, the first DRAM-free, IFC-based architecture that stores both model weights and KV cache entirely in compute-enabled 3D NAND flash. KVNAND addresses the fundamental performance challenges of flash under intensive KV cache access by leveraging IFC for all memory-bound operations to reduce data transfer overhead, introducing head-group parallelism to boost throughput, and employing page-level KV cache mapping to align token access patterns with flash organization. In addition, we propose a design space exploration framework that evaluates discrete and compact KVNAND variants to balance weight and KV placement, automatically identifying the optimal design trade-off. These techniques mitigate latency, energy, and reliability concerns, turning flash into a practical medium for long-context KV storage. Evaluations on MHA 7B and GQA 70B LLMs show that KVNAND achieves 1.98\(\times\)/1.94\(\times\)/2.05\(\times\) geomean speedup at 128/1K/10K-token contexts compared to DRAM-equipped IFC designs and addresses out-of-memory failures at 100K context length.