FB-RAG: Improving RAG with Forward and Backward Lookup

TL;DR

FB-RAG introduces a forward-backward lookup strategy that enhances retrieval-augmented generation by leveraging future generation insights, leading to improved accuracy and reduced latency without complex fine-tuning.

Contribution

The paper presents a training-free, forward-looking framework for RAG that improves relevance and efficiency by using evidence from multiple outputs to guide context selection.

Findings

Consistently outperforms baselines across 9 datasets.

Achieves over 48% latency reduction on EN.QA.

Guides final model effectively even when forward-looking LLM fails.

Abstract

Traditional Retrieval-Augmented Generation (RAG) struggles with complex queries that lack strong signals to retrieve the most relevant context, forcing a trade-off between choosing a small context that misses key information and a large context that confuses the LLM. To address this, we propose Forward-Backward RAG (FB-RAG), a new training-free framework based on a simple yet powerful forward-looking strategy. FB-RAG employs a light-weight LLM to peek into potential future generations, using evidence from multiple sampled outputs to precisely identify the most relevant context for a final, more powerful generator. This improves performance without complex finetuning or Reinforcement Learning common in prior work. Across $9$ datasets from LongBench and $\infty$Bench, FB-RAG consistently delivers strong results. Further, the performance gains can be achieved with reduced latency due to a shorter, more focused prompt for the powerful generator. On EN.QA dataset, FB-RAG matches the leading baseline with over $48$% latency reduction or achieves an $8$% performance improvement with a $10$% latency reduction. Our analysis finds cases where even when the forward-looking LLM fails to generate correct answers, its attempts are sufficient to guide the final model to an accurate response, demonstrating how smaller LLMs can systematically improve the performance and efficiency of larger ones.