RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems

TL;DR

RAGPerf is a comprehensive benchmarking framework that evaluates retrieval-augmented generation systems by modularly analyzing components, supporting diverse datasets, and automating performance and accuracy metrics.

Contribution

It introduces a modular, configurable benchmarking framework for RAG systems, enabling detailed performance profiling and analysis across various components and real-world scenarios.

Findings

RAGPerf incurs negligible performance overhead.

Supports diverse datasets and retrieval models.

Provides detailed metrics for system behavior analysis.

Abstract

We present the design and implementation of a RAG-based AI system benchmarking (RAGPerf) framework for characterizing the system behaviors of RAG pipelines. To facilitate detailed profiling and fine-grained performance analysis, RAGPerf decouples the RAG workflow into several modular components - embedding, indexing, retrieval, reranking, and generation. RAGPerf offers the flexibility for users to configure the core parameters of each component and examine their impact on the end-to-end query performance and quality. RAGPerf has a workload generator to model real-world scenarios by supporting diverse datasets (e.g., text, pdf, code, and audio), different retrieval and update ratios, and query distributions. RAGPerf also supports different embedding models, major vector databases such as LanceDB, Milvus, Qdrant, Chroma, and Elasticsearch, as well as different LLMs for content generation. It automates the collection of performance metrics (i.e., end-to-end query throughput, host/GPU memory footprint, and CPU/GPU utilization) and accuracy metrics (i.e., context recall, query accuracy, and factual consistency). We demonstrate the capabilities of RAGPerf through a comprehensive set of experiments and open source its codebase at GitHub. Our evaluation shows that RAGPerf incurs negligible performance overhead.