Scaling Retrieval Augmented Generation with RAG Fusion: Lessons from an Industry Deployment

TL;DR

This paper evaluates retrieval fusion techniques in a real-world RAG system, finding that increased recall from fusion does not translate into better answer quality under production constraints, highlighting the need for holistic evaluation.

Contribution

It provides an empirical assessment of retrieval fusion methods in a production RAG pipeline, revealing their limited benefits under realistic constraints and emphasizing the importance of comprehensive evaluation.

Findings

Retrieval fusion increases raw recall but is neutralized after re-ranking.

Fusion variants do not outperform single-query baselines on KB-level Top-$k$ accuracy.

Fusion introduces latency without improving downstream effectiveness.

Abstract

Retrieval-Augmented Generation (RAG) systems commonly adopt retrieval fusion techniques such as multi-query retrieval and reciprocal rank fusion (RRF) to increase document recall, under the assumption that higher recall leads to better answer quality. While these methods show consistent gains in isolated retrieval benchmarks, their effectiveness under realistic production constraints remains underexplored. In this work, we evaluate retrieval fusion in a production-style RAG pipeline operating over an enterprise knowledge base, with fixed retrieval depth, re-ranking budgets, and latency constraints. Across multiple fusion configurations, we find that retrieval fusion does increase raw recall, but these gains are largely neutralized after re-ranking and truncation. In our setting, fusion variants fail to outperform single-query baselines on KB-level Top-$k$ accuracy, with Hit@10 decreasing from $0.51$ to $0.48$ in several configurations. Moreover, fusion introduces additional latency overhead due to query rewriting and larger candidate sets, without corresponding improvements in downstream effectiveness. Our analysis suggests that recall-oriented fusion techniques exhibit diminishing returns once realistic re-ranking limits and context budgets are applied. We conclude that retrieval-level improvements do not reliably translate into end-to-end gains in production RAG systems, and argue for evaluation frameworks that jointly consider retrieval quality, system efficiency, and downstream impact.