Constructing and Evaluating Declarative RAG Pipelines in PyTerrier

TL;DR

This paper presents a framework for constructing and evaluating declarative RAG pipelines within PyTerrier, enabling efficient integration of retrieval and generation components for improved search and question-answering systems.

Contribution

It introduces PyTerrier-RAG, a new extension that simplifies building and assessing RAG pipelines using declarative syntax and integrates with various retrieval models and datasets.

Findings

Demonstrates succinct pipeline construction on standard datasets

Shows integration with state-of-the-art retrievers and neural rankers

Highlights advantages of declarative architecture in RAG pipelines

Abstract

Search engines often follow a pipeline architecture, where complex but effective reranking components are used to refine the results of an initial retrieval. Retrieval augmented generation (RAG) is an exciting application of the pipeline architecture, where the final component generates a coherent answer for the users from the retrieved documents. In this demo paper, we describe how such RAG pipelines can be formulated in the declarative PyTerrier architecture, and the advantages of doing so. Our PyTerrier-RAG extension for PyTerrier provides easy access to standard RAG datasets and evaluation measures, state-of-the-art LLM readers, and using PyTerrier's unique operator notation, easy-to-build pipelines. We demonstrate the succinctness of indexing and RAG pipelines on standard datasets (including Natural Questions) and how to build on the larger PyTerrier ecosystem with state-of-the-art sparse, learned-sparse, and dense retrievers, and other neural rankers.