AMAQA: A Metadata-based QA Dataset for RAG Systems
Davide Bruni, Marco Avvenuti, Nicola Tonellotto, Maurizio Tesconi

TL;DR
AMAQA is a novel open-access QA dataset that integrates metadata with textual data, significantly improving the evaluation and performance of RAG systems in scenarios requiring external information and rapid data analysis.
Contribution
This paper introduces AMAQA, the first single-hop QA benchmark incorporating metadata, and demonstrates how leveraging metadata enhances RAG system accuracy.
Findings
Metadata boosts GPT-4o accuracy from 0.5 to 0.86.
Metadata improves open source LLM accuracy from 0.27 to 0.76.
Extensive tests set new benchmarks for metadata-driven QA.
Abstract
Retrieval-augmented generation (RAG) systems are widely used in question-answering (QA) tasks, but current benchmarks lack metadata integration, limiting their evaluation in scenarios requiring both textual data and external information. To address this, we present AMAQA, a new open-access QA dataset designed to evaluate tasks combining text and metadata. The integration of metadata is especially important in fields that require rapid analysis of large volumes of data, such as cybersecurity and intelligence, where timely access to relevant information is critical. AMAQA includes about 1.1 million English messages collected from 26 public Telegram groups, enriched with metadata such as timestamps and chat names. It also contains 20,000 hotel reviews with metadata. In addition, the dataset provides 2,600 high-quality QA pairs built across both domains, Telegram messages and hotel reviews,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFault Detection and Control Systems · AI-based Problem Solving and Planning
MethodsAttention Is All You Need · Linear Warmup With Linear Decay · Softmax · Attention Dropout · WordPiece · Refunds@Expedia|||How do I get a full refund from Expedia? · Linear Layer · Residual Connection · Byte Pair Encoding · Weight Decay
