Efficient Test-Time Retrieval Augmented Generation

TL;DR

ET2RAG is a training-free, efficient retrieval-augmented framework that enhances large language models' accuracy by retrieving relevant documents, generating diverse responses, and selecting the best via majority voting, balancing performance and computational cost.

Contribution

The paper introduces ET2RAG, a novel training-free method that improves LLM performance using retrieval, partial generation, and majority voting to balance efficiency and accuracy.

Findings

Significantly improves performance on question answering, recipe generation, and image captioning.

Reduces computational cost by using partial responses for consensus.

Effective across multiple tasks without additional training.

Abstract

Although Large Language Models (LLMs) demonstrate significant capabilities, their reliance on parametric knowledge often leads to inaccuracies. Retrieval Augmented Generation (RAG) mitigates this by incorporating external knowledge, but these methods may introduce irrelevant retrieved documents, leading to inaccurate responses. While the integration methods filter out incorrect answers from multiple responses, but lack external knowledge like RAG methods, and their high costs require balancing overhead with performance gains. To address these issues, we propose an Efficient Test-Time Retrieval-Augmented Generation Framework named ET2RAG to improve the performance of LLMs while maintaining efficiency. Specifically, ET2RAG is a training-free method, that first retrieves the most relevant documents and augments the LLMs to efficiently generate diverse candidate responses by managing response length. Then we compute the similarity of candidate responses and employ a majority voting mechanism to select the most suitable response as the final output. In particular, we discover that partial generation is sufficient to capture the key information necessary for consensus calculation, allowing us to effectively perform majority voting without the need for fully generated responses. Thus, we can reach a balance between computational cost and performance by managing the response length for the number of retrieved documents for majority voting. Experimental results demonstrate that ET2RAG significantly enhances performance across three tasks, including open-domain question answering, recipe generation and image captioning.