FedMosaic: Federated Retrieval-Augmented Generation via Parametric Adapters

TL;DR

FedMosaic introduces a federated retrieval-augmented generation framework using parametric adapters, enabling privacy-preserving knowledge grounding with reduced storage and communication costs, and improved accuracy.

Contribution

It is the first federated RAG framework utilizing parametric adapters with clustering and selective aggregation to enhance efficiency and privacy.

Findings

Achieves 10.9% higher accuracy than state-of-the-art methods.

Reduces storage costs by up to 86.3%.

Reduces communication costs by up to 91.4%.

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding generation in external knowledge to improve factuality and reduce hallucinations. Yet most deployments assume a centralized corpus, which is infeasible in privacy aware domains where knowledge remains siloed. This motivates federated RAG (FedRAG), where a central LLM server collaborates with distributed silos without sharing raw documents. In context RAG violates this requirement by transmitting verbatim documents, whereas parametric RAG encodes documents into lightweight adapters that merge with a frozen LLM at inference, avoiding raw-text exchange. We adopt the parametric approach but face two unique challenges induced by FedRAG: high storage and communication from per-document adapters, and destructive aggregation caused by indiscriminately merging multiple adapters. We present FedMosaic, the first federated RAG framework built on parametric adapters. FedMosaic clusters semantically related documents into multi-document adapters with document-specific masks to reduce overhead while preserving specificity, and performs selective adapter aggregation to combine only relevance-aligned, nonconflicting adapters. Experiments show that FedMosaic achieves an average 10.9% higher accuracy than state-of-the-art methods in four categories, while lowering storage costs by 78.8% to 86.3% and communication costs by 91.4%, and never sharing raw documents.