Parallel Key-Value Cache Fusion for Position Invariant RAG
Philhoon Oh, Jinwoo Shin, James Thorne

TL;DR
This paper introduces a framework for decoder-only models in RAG that achieves position invariance, ensuring consistent outputs regardless of context order, and demonstrates improved robustness in open domain question answering tasks.
Contribution
The proposed framework enables position-invariant RAG for decoder-only models, addressing the 'Lost in the Middle' problem and enhancing robustness against irrelevant passages.
Findings
Achieves position invariance in RAG models
Demonstrates superior robustness in question answering tasks
Outperforms existing approaches in handling irrelevant passages
Abstract
Recent advancements in Large Language Models (LLMs) underscore the necessity of Retrieval Augmented Generation (RAG) to leverage external information. However, LLMs are sensitive to the position of relevant information within contexts and tend to generate incorrect responses when such information is placed in the middle, known as `Lost in the Middle' phenomenon. In this paper, we introduce a framework that generates consistent outputs for decoder-only models, irrespective of the input context order. Experimental results for three open domain question answering tasks demonstrate position invariance, where the model is not sensitive to input context order, and superior robustness to irrelevent passages compared to prevailing approaches for RAG pipelines.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAlgorithms and Data Compression · Parallel Computing and Optimization Techniques
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Layer Normalization · Dense Connections · Linear Warmup With Linear Decay · WordPiece · Attention Dropout · Adam · Residual Connection · Dropout · Softmax
