Towards a RAG-based Summarization Agent for the Electron-Ion Collider

TL;DR

This paper presents RAGS4EIC, a retrieval-augmented summarization AI for the Electron-Ion Collider that combines vector database querying and large language models to generate concise, referenced summaries of complex experimental data, enhancing collaboration.

Contribution

The paper introduces a novel RAG-based summarization framework tailored for the EIC community, integrating prompt-tuning, LangChain, and a web app for scalable, accurate data summarization.

Findings

Effective summarization with citation referencing

Flexible prompt instruction-tuning improves accuracy

Web app demonstrates practical deployment

Abstract

The complexity and sheer volume of information encompassing documents, papers, data, and other resources from large-scale experiments demand significant time and effort to navigate, making the task of accessing and utilizing these varied forms of information daunting, particularly for new collaborators and early-career scientists. To tackle this issue, a Retrieval Augmented Generation (RAG)--based Summarization AI for EIC (RAGS4EIC) is under development. This AI-Agent not only condenses information but also effectively references relevant responses, offering substantial advantages for collaborators. Our project involves a two-step approach: first, querying a comprehensive vector database containing all pertinent experiment information; second, utilizing a Large Language Model (LLM) to generate concise summaries enriched with citations based on user queries and retrieved data. We describe the evaluation methods that use RAG assessments (RAGAs) scoring mechanisms to assess the effectiveness of responses. Furthermore, we describe the concept of prompt template-based instruction-tuning which provides flexibility and accuracy in summarization. Importantly, the implementation relies on LangChain, which serves as the foundation of our entire workflow. This integration ensures efficiency and scalability, facilitating smooth deployment and accessibility for various user groups within the Electron Ion Collider (EIC) community. This innovative AI-driven framework not only simplifies the understanding of vast datasets but also encourages collaborative participation, thereby empowering researchers. As a demonstration, a web application has been developed to explain each stage of the RAG Agent development in detail.