Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature

TL;DR

Clinfo.ai is an open-source system that retrieves and summarizes scientific literature to answer medical questions, addressing evaluation challenges and providing a new dataset and benchmark for such tools.

Contribution

The paper introduces Clinfo.ai, a retrieval-augmented LLM system for medical questions, along with a new dataset and evaluation benchmark for scientific literature summarization.

Findings

Clinfo.ai effectively retrieves relevant scientific literature.

Benchmark results highlight strengths and weaknesses of current systems.

The PubMedRS-200 dataset enables systematic evaluation of medical QA tools.

Abstract

The quickly-expanding nature of published medical literature makes it challenging for clinicians and researchers to keep up with and summarize recent, relevant findings in a timely manner. While several closed-source summarization tools based on large language models (LLMs) now exist, rigorous and systematic evaluations of their outputs are lacking. Furthermore, there is a paucity of high-quality datasets and appropriate benchmark tasks with which to evaluate these tools. We address these issues with four contributions: we release Clinfo.ai, an open-source WebApp that answers clinical questions based on dynamically retrieved scientific literature; we specify an information retrieval and abstractive summarization task to evaluate the performance of such retrieval-augmented LLM systems; we release a dataset of 200 questions and corresponding answers derived from published systematic reviews, which we name PubMed Retrieval and Synthesis (PubMedRS-200); and report benchmark results for Clinfo.ai and other publicly available OpenQA systems on PubMedRS-200.