Meta Knowledge for Retrieval Augmented Large Language Models

TL;DR

This paper introduces a new data-centric retrieval augmented generation framework that uses meta knowledge and synthetic QA to improve information retrieval and synthesis in large language models, achieving higher accuracy and depth.

Contribution

The paper presents a novel prepare-then-rewrite-then-retrieve framework with meta knowledge summaries, enhancing RAG systems with synthetic QA and metadata for better domain understanding.

Findings

Augmented queries with synthetic question matching outperform traditional RAG (p < 0.01)

Meta knowledge-augmented queries improve retrieval precision and answer quality

Cost-effective approach using Claude 3 Haiku for large-scale document processing

Abstract

Retrieval Augmented Generation (RAG) is a technique used to augment Large Language Models (LLMs) with contextually relevant, time-critical, or domain-specific information without altering the underlying model parameters. However, constructing RAG systems that can effectively synthesize information from large and diverse set of documents remains a significant challenge. We introduce a novel data-centric RAG workflow for LLMs, transforming the traditional retrieve-then-read system into a more advanced prepare-then-rewrite-then-retrieve-then-read framework, to achieve higher domain expert-level understanding of the knowledge base. Our methodology relies on generating metadata and synthetic Questions and Answers (QA) for each document, as well as introducing the new concept of Meta Knowledge Summary (MK Summary) for metadata-based clusters of documents. The proposed innovations enable personalized user-query augmentation and in-depth information retrieval across the knowledge base. Our research makes two significant contributions: using LLMs as evaluators and employing new comparative performance metrics, we demonstrate that (1) using augmented queries with synthetic question matching significantly outperforms traditional RAG pipelines that rely on document chunking (p < 0.01), and (2) meta knowledge-augmented queries additionally significantly improve retrieval precision and recall, as well as the final answers breadth, depth, relevancy, and specificity. Our methodology is cost-effective, costing less than $20 per 2000 research papers using Claude 3 Haiku, and can be adapted with any fine-tuning of either the language or embedding models to further enhance the performance of end-to-end RAG pipelines.