Multi-task retriever fine-tuning for domain-specific and efficient RAG

TL;DR

This paper proposes a multi-task instruction fine-tuning approach for retriever encoders in RAG systems, enabling scalable, low-cost, and effective domain-specific retrieval across multiple real-world applications.

Contribution

It introduces a multi-task fine-tuning method for retriever encoders that generalizes across domains and tasks, reducing the need for multiple retrievers and lowering deployment costs.

Findings

The multi-task fine-tuned retriever encoder performs well on out-of-domain data.

It effectively handles unseen retrieval tasks in real-world enterprise scenarios.

The approach improves scalability and efficiency of RAG systems.

Abstract

Retrieval-Augmented Generation (RAG) has become ubiquitous when deploying Large Language Models (LLMs), as it can address typical limitations such as generating hallucinated or outdated information. However, when building real-world RAG applications, practical issues arise. First, the retrieved information is generally domain-specific. Since it is computationally expensive to fine-tune LLMs, it is more feasible to fine-tune the retriever to improve the quality of the data included in the LLM input. Second, as more applications are deployed in the same real-world system, one cannot afford to deploy separate retrievers. Moreover, these RAG applications normally retrieve different kinds of data. Our solution is to instruction fine-tune a small retriever encoder on a variety of domain-specific tasks to allow us to deploy one encoder that can serve many use cases, thereby achieving low-cost, scalability, and speed. We show how this encoder generalizes to out-of-domain settings as well as to an unseen retrieval task on real-world enterprise use cases.