Retrieval Augmented Generation Evaluation for Health Documents

TL;DR

This paper evaluates Retrieval Augmented Generation (RAG) methods for processing healthcare documents, introducing a new pipeline, evaluation tools, and a benchmark dataset to improve accuracy and trustworthiness in medical information synthesis.

Contribution

It presents RAGEv, a novel pipeline with evaluation tools and a benchmark dataset for assessing RAG approaches in health document analysis, highlighting their potential and challenges.

Findings

High accuracy in short yes/no and long answers

Careful implementation reduces common LLM issues in healthcare

Potential for integration into policy support tasks

Abstract

Safe and trustworthy use of Large Language Models (LLM) in the processing of healthcare documents and scientific papers could substantially help clinicians, scientists and policymakers in overcoming information overload and focusing on the most relevant information at a given moment. Retrieval Augmented Generation (RAG) is a promising method to leverage the potential of LLMs while enhancing the accuracy of their outcomes. This report assesses the potentials and shortcomings of such approaches in the automatic knowledge synthesis of different types of documents in the health domain. To this end, it describes: (1) an internally developed proof of concept pipeline that employs state-of-the-art practices to deliver safe and trustable analysis for healthcare documents and scientific papers called RAGEv (Retrieval Augmented Generation Evaluation); (2) a set of evaluation tools for LLM-based document retrieval and generation; (3) a benchmark dataset to verify the accuracy and veracity of the results called RAGEv-Bench. It concludes that careful implementations of RAG techniques could minimize most of the common problems in the use of LLMs for document processing in the health domain, obtaining very high scores both on short yes/no answers and long answers. There is a high potential for incorporating it into the day-to-day work of policy support tasks, but additional efforts are required to obtain a consistent and trustworthy tool.