SEARA: An Automated Approach for Obtaining Optimal Retrievers

TL;DR

SEARA introduces an efficient, automated method for evaluating and optimizing retrievers in Retrieval-Augmented Generation systems, reducing costs and improving domain-specific performance assessment.

Contribution

The paper presents SEARA, a novel subset sampling evaluation technique that enables low-cost, automated retrieval system assessment tailored to specific applications.

Findings

Successfully applied to knowledge-based Q&A and travel assistant scenarios.

Achieved robust retrieval evaluation with minimal data and comprehensive metrics.

Enabled automatic selection of optimal retrievers for different domains.

Abstract

Retrieval-Augmented Generation (RAG) is a core approach for enhancing Large Language Models (LLMs), where the effectiveness of the retriever largely determines the overall response quality of RAG systems. Retrievers encompass a multitude of hyperparameters that significantly impact performance outcomes and demonstrate sensitivity to specific applications. Nevertheless, hyperparameter optimization entails prohibitively high computational expenses. Existing evaluation methods suffer from either prohibitive costs or disconnection from domain-specific scenarios. This paper proposes SEARA (Subset sampling Evaluation for Automatic Retriever Assessment), which addresses evaluation data challenges through subset sampling techniques and achieves robust automated retriever evaluation by minimal retrieval facts extraction and comprehensive retrieval metrics. Based on real user queries, this method enables fully automated retriever evaluation at low cost, thereby obtaining optimal retriever for specific business scenarios. We validate our method across classic RAG applications in rednote, including knowledge-based Q\&A system and retrieval-based travel assistant, successfully obtaining scenario-specific optimal retrievers.