AMAQA: A Metadata-based QA Dataset for RAG Systems

TL;DR

AMAQA is a novel open-access QA dataset that integrates metadata with textual data, significantly improving the evaluation and performance of RAG systems in scenarios requiring external information and rapid data analysis.

Contribution

This paper introduces AMAQA, the first single-hop QA benchmark incorporating metadata, and demonstrates how leveraging metadata enhances RAG system accuracy.

Findings

Metadata boosts GPT-4o accuracy from 0.5 to 0.86.

Metadata improves open source LLM accuracy from 0.27 to 0.76.

Extensive tests set new benchmarks for metadata-driven QA.

Abstract

Retrieval-augmented generation (RAG) systems are widely used in question-answering (QA) tasks, but current benchmarks lack metadata integration, limiting their evaluation in scenarios requiring both textual data and external information. To address this, we present AMAQA, a new open-access QA dataset designed to evaluate tasks combining text and metadata. The integration of metadata is especially important in fields that require rapid analysis of large volumes of data, such as cybersecurity and intelligence, where timely access to relevant information is critical. AMAQA includes about 1.1 million English messages collected from 26 public Telegram groups, enriched with metadata such as timestamps and chat names. It also contains 20,000 hotel reviews with metadata. In addition, the dataset provides 2,600 high-quality QA pairs built across both domains, Telegram messages and hotel reviews, making AMAQA a valuable resource for advancing research on metadata-driven QA and RAG systems. Both Telegram messages and Hotel reviews are enriched with emotional tones or toxicity indicators. To the best of our knowledge, AMAQA is the first single-hop QA benchmark to incorporate metadata. We conduct extensive tests on the benchmark, setting a new reference point for future research. We show that leveraging metadata boosts accuracy from 0.5 to 0.86 for GPT-4o and from 0.27 to 0.76 for open source LLMs, highlighting the value of structured context. We conducted experiments on our benchmark to assess the performance of known techniques designed to enhance RAG, highlighting the importance of properly managing metadata throughout the entire RAG pipeline.