ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search

TL;DR

ReSLLM leverages large language models for effective zero-shot resource selection in federated search, enhanced further through an unsupervised fine-tuning protocol called SLAT, improving efficiency and relevance without extensive labeled data.

Contribution

The paper introduces ReSLLM, a novel zero-shot resource selection method using LLMs, and proposes SLAT, an unsupervised fine-tuning protocol that boosts performance without labeled training data.

Findings

ReSLLM achieves competitive zero-shot resource selection performance.

SLAT fine-tuning significantly improves ReSLLM effectiveness.

ReSLLM reduces reliance on labor-intensive labeled datasets.

Abstract

Federated search, which involves integrating results from multiple independent search engines, will become increasingly pivotal in the context of Retrieval-Augmented Generation pipelines empowering LLM-based applications such as chatbots. These systems often distribute queries among various search engines, ranging from specialized (e.g., PubMed) to general (e.g., Google), based on the nature of user utterances. A critical aspect of federated search is resource selection - the selection of appropriate resources prior to issuing the query to ensure high-quality and rapid responses, and contain costs associated with calling the external search engines. However, current SOTA resource selection methodologies primarily rely on feature-based learning approaches. These methods often involve the labour intensive and expensive creation of training labels for each resource. In contrast, LLMs have exhibited strong effectiveness as zero-shot methods across NLP and IR tasks. We hypothesise that in the context of federated search LLMs can assess the relevance of resources without the need for extensive predefined labels or features. In this paper, we propose ReSLLM. Our ReSLLM method exploits LLMs to drive the selection of resources in federated search in a zero-shot setting. In addition, we devise an unsupervised fine tuning protocol, the Synthetic Label Augmentation Tuning (SLAT), where the relevance of previously logged queries and snippets from resources is predicted using an off-the-shelf LLM and then in turn used to fine-tune ReSLLM with respect to resource selection. Our empirical evaluation and analysis details the factors influencing the effectiveness of LLMs in this context. The results showcase the merits of ReSLLM for resource selection: not only competitive effectiveness in the zero-shot setting, but also obtaining large when fine-tuned using SLAT-protocol.