HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement

TL;DR

This paper introduces HA-RAG, a hotness-aware inference system for Retrieval-Augmented Generation that reduces memory and latency by using mixed precision and strategic data placement based on access frequency.

Contribution

It proposes a novel hotness-aware data compression and placement strategy to accelerate RAG inference with minimal accuracy impact.

Findings

Achieves an average of 2.10x speedup in TTFT.

Maximum speedup of 10.49x in TTFT.

Negligible accuracy loss compared to TurboRAG.

Abstract

Retrieval-Augmented Generation (RAG) improves model output accuracy by leveraging external knowledge bases, serving as an effective solution to address hallucination issues and knowledge-update delays in Large Language Models (LLMs). However, the introduction of external knowledge bases presents RAG with challenges in long-context processing, significantly increasing memory consumption and inference latency. Existing research accelerates inference by precomputing Key and Value (KV) of the knowledge base and loading them on-demand during inference. Based on the access frequency of different KV chunks within the external knowledge base, this paper proposes a hotness-aware RAG (HA-RAG) inference optimization system. First, leveraging the numerical distribution of KV chunks, we introduce a hotness-aware mixed-precision compressing and loading method to reduce disk I/O and memory access overhead. Second, we design a hotness-aware data placement strategy that prioritizes storing frequently accessed KV chunks in high-speed memory to improve data access efficiency. Experimental results demonstrate that, compared with TurboRAG, the proposed HA-RAG achieves an average speedup of 2.10x and maximum speedup of 10.49x in Time-To-First-Token (TTFT) with negligible accuracy loss.