Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
Tristan Kenneweg, Philip Kenneweg, Barbara Hammer

TL;DR
This paper introduces a rigorous workflow for creating datasets and evaluating Retrieval Augmented Generation (RAG) strategies, enabling quantitative comparison and development of sophisticated RAG systems like boolean agents that optimize database queries.
Contribution
It presents a novel dataset creation and evaluation methodology for RAG systems and demonstrates its use in developing a boolean agent RAG setup that improves efficiency.
Findings
Established a quantitative comparison framework for RAG strategies.
Developed a boolean agent RAG system that reduces unnecessary database queries.
Published code and dataset for community use.
Abstract
Retrieval Augmented Generation (RAG) systems have seen huge popularity in augmenting Large-Language Model (LLM) outputs with domain specific and time sensitive data. Very recently a shift is happening from simple RAG setups that query a vector database for additional information with every user input to more sophisticated forms of RAG. However, different concrete approaches compete on mostly anecdotal evidence at the moment. In this paper we present a rigorous dataset creation and evaluation workflow to quantitatively compare different RAG strategies. We use a dataset created this way for the development and evaluation of a boolean agent RAG setup: A system in which a LLM can decide whether to query a vector database or not, thus saving tokens on questions that can be answered with internal knowledge. We publish our code and generated dataset online.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEnergy Load and Power Forecasting · Power Systems and Technologies · Smart Grid Energy Management
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Linear Layer · WordPiece · Layer Normalization · Byte Pair Encoding · Dropout · Multi-Head Attention · Attention Dropout · Linear Warmup With Linear Decay
