RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generation

TL;DR

RoutIR is a Python package that enables fast, flexible, and online serving of complex retrieval pipelines for Retrieval-Augmented Generation systems, supporting dynamic configurations and multiple retrieval methods.

Contribution

The paper introduces RoutIR, a novel Python package that facilitates real-time, customizable retrieval pipelines with an easy-to-use HTTP API for RAG systems.

Findings

Supports arbitrary retrieval methods including reranking and query expansion.

Enables dynamic, on-the-fly construction of retrieval pipelines.

Provides asynchronous batching and caching for efficiency.

Abstract

Retrieval models are key components of Retrieval-Augmented Generation (RAG) systems, which generate search queries, process the documents returned, and generate a response. RAG systems are often dynamic and may involve multiple rounds of retrieval. While many state-of-the-art retrieval methods are available through academic IR platforms, these platforms are typically designed for the Cranfield paradigm in which all queries are known up front and can be batch processed offline. This simplification accelerates research but leaves state-of-the-art retrieval models unable to support downstream applications that require online services, such as arbitrary dynamic RAG pipelines that involve looping, feedback, or even self-organizing agents. In this work, we introduce RoutIR, a Python package that provides a simple and efficient HTTP API that wraps arbitrary retrieval methods, including first stage retrieval, reranking, query expansion, and result fusion. By providing a minimal JSON configuration file specifying the retrieval models to serve, RoutIR can be used to construct and query retrieval pipelines on-the-fly using any permutation of available models (e.g., fusing the results of several first-stage retrieval methods followed by reranking). The API automatically performs asynchronous query batching and caches results by default. While many state-of-the-art retrieval methods are already supported by the package, RoutIR is also easily expandable by implementing the Engine abstract class. The package is open-sourced and publicly available on GitHub: http://github.com/hltcoe/routir.