ReBOL: Retrieval via Bayesian Optimization with Batched LLM Relevance Observations and Query Reformulation

TL;DR

ReBOL introduces a Bayesian Optimization-based retrieval method that leverages LLM query reformulation and batch relevance scoring to improve recall and ranking in document retrieval tasks, addressing limitations of traditional reranking approaches.

Contribution

This paper presents ReBOL, a novel retrieval framework combining LLM reformulation with Bayesian Optimization to enhance top-k document retrieval performance.

Findings

ReBOL outperforms baseline LLM rerankers in recall and NDCG metrics.

ReBOL achieves comparable latency to existing reranking methods.

ReBOL demonstrates consistent improvements across multiple datasets and LLMs.

Abstract

LLM-reranking is limited by the top-k documents retrieved by vector similarity, which neither enables contextual query-document token interactions nor captures multimodal relevance distributions. While LLM query reformulation attempts to improve recall by generating improved or additional queries, it is still followed by vector similarity retrieval. We thus propose to address these top-k retrieval stage failures by introducing ReBOL, which 1) uses LLM query reformulations to initialize a multimodal Bayesian Optimization (BO) posterior over document relevance, and 2) iteratively acquires document batches for LLM query-document relevance scoring followed by posterior updates to optimize relevance. After exploring query reformulation and document batch diversification techniques, we evaluate ReBOL against LLM reranker baselines on five BEIR datasets and using two LLMs (Gemini-2.5-Flash-Lite, GPT-5.2). ReBOL consistently achieves higher recall and competitive rankings, for example compared to the best LLM reranker on the Robust04 dataset with 46.5% vs. 35.0% recall@100 and 63.6% vs. 61.2% NDCG@10. We also show that ReBOL can achieve comparable latency to LLM rerankers.