PentaRAG: Large-Scale Intelligent Knowledge Retrieval for Enterprise LLM Applications

TL;DR

PentaRAG is a five-layer retrieval system that enhances enterprise LLM applications by improving speed, accuracy, and resource efficiency through layered caching, memory recall, and retrieval strategies.

Contribution

The paper introduces PentaRAG, a novel five-layer module that significantly improves retrieval speed, accuracy, and resource efficiency in enterprise LLM deployments compared to traditional RAG systems.

Findings

Reduces latency from seconds to under one second.

Increases answer similarity by 8% and factual correctness by 16%.

Achieves approximately 100,000 queries per second throughput.

Abstract

Enterprise deployments of large-language model (LLM) demand continuously changing document collections with sub-second latency and predictable GPU cost requirements that classical Retrieval-Augmented Generation (RAG) pipelines only partially satisfy. We present PentaRAG, a five-layer module that routes each query through two instant caches (fixed key-value and semantic), a memory-recall mode that exploits the LLM's own weights, an adaptive session memory, and a conventional retrieval-augmentation layer. Implemented with Mistral-8B, Milvus and vLLM, the system can answer most repeated or semantically similar questions from low-latency caches while retaining full retrieval for novel queries. On the TriviaQA domain, LoRA fine-tuning combined with the memory-recall layer raises answer similarity by approximately 8% and factual correctness by approximately 16% over the base model. Under a nine-session runtime simulation, cache warming reduces mean latency from several seconds to well below one second and shifts traffic toward the fast paths. Resource-efficiency tests show that PentaRAG cuts average GPU time to 0.248 seconds per query, roughly half that of a naive RAG baseline, and sustains an aggregate throughput of approximately 100,000 queries per second on our setup. These results demonstrate that a layered routing strategy can deliver freshness, speed, and efficiency simultaneously in production-grade RAG systems.