DIRC-RAG: Accelerating Edge RAG with Robust High-Density and High-Loading-Bandwidth Digital In-ReRAM Computation

TL;DR

DIRC-RAG introduces a high-density, low-power digital in-ReRAM computation architecture to accelerate retrieval-augmented generation on edge devices, significantly reducing latency and energy consumption while maintaining accuracy.

Contribution

It presents a novel DIRC architecture combining high-density ReRAM with digital MAC operations, enabling efficient, robust, and low-power edge RAG acceleration.

Findings

Achieves 5.18Mb/mm2 memory density with 131 TOPS throughput.

Reduces retrieval latency to 5.6μs/query and energy to 0.956μJ/query.

Maintains retrieval precision with error optimization and detection circuits.

Abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge retrieval but faces challenges on edge devices due to high storage, energy, and latency demands. Computing-in-Memory (CIM) offers a promising solution by storing document embeddings in CIM macros and enabling in-situ parallel retrievals but is constrained by either low memory density or limited computational accuracy. To address these challenges, we present DIRCRAG, a novel edge RAG acceleration architecture leveraging Digital In-ReRAM Computation (DIRC). DIRC integrates a high-density multi-level ReRAM subarray with an SRAM cell, utilizing SRAM and differential sensing for robust ReRAM readout and digital multiply-accumulate (MAC) operations. By storing all document embeddings within the CIM macro, DIRC achieves ultra-low-power, single-cycle data loading, substantially reducing both energy consumption and latency compared to offchip DRAM. A query-stationary (QS) dataflow is supported for RAG tasks, minimizing on-chip data movement and reducing SRAM buffer requirements. We introduce error optimization for the DIRC ReRAM-SRAM cell by extracting the bit-wise spatial error distribution of the ReRAM subarray and applying targeted bit-wise data remapping. An error detection circuit is also implemented to enhance readout resilience against deviceand circuit-level variations. Simulation results demonstrate that DIRC-RAG under TSMC40nm process achieves an on-chip non-volatile memory density of 5.18Mb/mm2 and a throughput of 131 TOPS. It delivers a 4MB retrieval latency of 5.6{\mu}s/query and an energy consumption of 0.956{\mu}J/query, while maintaining the retrieval precision.