A Decentralized Retrieval Augmented Generation System with Source Reliabilities Secured on Blockchain

TL;DR

This paper introduces a decentralized retrieval-augmented generation system that uses blockchain to securely evaluate source reliability, improving response quality and reducing costs compared to centralized systems.

Contribution

It presents a novel decentralized RAG architecture with blockchain-based reliability scoring, enhancing trustworthiness and performance in unreliable data environments.

Findings

Achieves +10.7% performance over centralized systems in unreliable data environments.

Approaches upper-bound performance in reliable data scenarios.

Reduces marginal costs by approximately 56% through batched updates.

Abstract

Existing retrieval-augmented generation (RAG) systems typically use a centralized architecture, causing a high cost of data collection, integration, and management, as well as privacy concerns. There is a great need for a decentralized RAG system that enables foundation models to utilize information directly from data owners who maintain full control over their sources. However, decentralization brings a challenge: the numerous independent data sources vary significantly in reliability, which can diminish retrieval accuracy and response quality. To address this, our decentralized RAG system has a novel reliability scoring mechanism that dynamically evaluates each source based on the quality of responses it contributes to generate and prioritizes high-quality sources during retrieval. To ensure transparency and trust, the scoring process is securely managed through blockchain-based smart contracts, creating verifiable and tamper-proof reliability records without relying on a central authority. We evaluate our decentralized system with two Llama models (3B and 8B) in two simulated environments where six data sources have different levels of reliability. Our system achieves a +10.7\% performance improvement over its centralized counterpart in the real world-like unreliable data environments. Notably, it approaches the upper-bound performance of centralized systems under ideally reliable data environments. The decentralized infrastructure enables secure and trustworthy scoring management, achieving approximately 56\% marginal cost savings through batched update operations. Our code and system are open-sourced at github.com/yining610/Reliable-dRAG.