Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapters

TL;DR

This paper introduces Poly-PRAG, a novel retrieval-augmented generation method that uses a small set of shared LoRA adapters with a latent routing function, improving efficiency and reducing storage costs in knowledge integration.

Contribution

Poly-PRAG proposes a shared adapter and routing mechanism, enabling scalable and efficient knowledge encoding in retrieval-augmented generation models.

Findings

Outperforms existing PRAG baselines on four benchmarks.

Reduces storage requirements by sharing adapters across documents.

Enhances encoding efficiency with a latent routing function.

Abstract

Parametric Retrieval-Augmented Generation (PRAG) is a RAG approach that integrates external knowledge directly into model parameters using a LoRA adapter, aiming at reducing the inference cost compared to traditional RAG. However, current PRAG approaches adopt a \textit{one-to-one} document encoding scheme, using a dedicated LoRA adapter for each individual document. This scheme introduces two major limitations: 1) As the number of documents increases, there will be a prohibitive cost for training and storage. 2) The LoRA adapters may largely overlap due to the shared knowledge across documents, making the approach highly inefficient. To overcome these challenges, we propose the Poly-PRAG approach, which uses a small set of LoRA adapters that are able to encode more general knowledge. Each document can be encoded using a combination of them through a latent routing function. By jointly training the LoRA adapters and the latent routing function, each LoRA adapter is able to encode a shared part of the knowledge across documents, and the routing function can select the best combination of adapters for a document. Experimental results on four benchmarks demonstrate the effectiveness of the Poly-PRAG compared to other strong PRAG baselines. In addition, this approach reduces the storage requirement by avoiding the need to store a large number of LoRA adapters and offers a more efficient way to encode external knowledge into LLMs.