CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG

TL;DR

CDS4RAG introduces a cyclic dual-sequential hyperparameter optimization framework for RAG, improving convergence speed and performance by separately optimizing retriever and generator hyperparameters.

Contribution

It presents a novel cyclic dual-sequential formulation that distinguishes and optimizes retriever and generator hyperparameters iteratively, enhancing efficiency and effectiveness.

Findings

Boosts vanilla algorithms in 21 out of 24 cases.

Outperforms state-of-the-art algorithms in all tested cases.

Achieves up to 1.54x improvements in generation quality.

Abstract

Retrieval-Augmented Generation (RAG) is sensitive to the vast hyperparameters of the retriever and generator, yet optimizing them using given queries is a challenging task due to the complex interactions and expensive evaluation costs. Existing algorithms are ineffective and slow in convergence, since they often treat RAG as a monolithic black box or only optimize partial hyperparameters. In this paper, we propose CDS4RAG, a framework that optimizes the full RAG hyperparameters using given queries via a new cyclic dual-sequential formulation. CDS4RAG is special in the sense that it distinguishes the hyperparameters of the retriever and generator, cyclically optimizing them in turn. Such a paradigm allows us to design fine-grained within-cycle budget provision and expedite the optimization via cross-cycle seeding when optimizing the generator. CDS4RAG is also an algorithm-agnostic framework that can be paired with diverse general algorithms. Through experiments on four common benchmarks and two backbone LLMs, we reveal that CDS4RAG considerably boosts the vanilla algorithms in 21/24 cases while significantly outperforming state-of-the-art algorithms in all cases with up to 1.54x improvements of generation quality and better speedup.