Digital Pathway Curation (DPC): a pipeline able to assess the reproducibility, consensus and accuracy in biomedical search retrieval by comparing Gemini, PubMed, and Scientific Reviewers

TL;DR

The paper introduces the Digital Pathway Curation pipeline to evaluate the reproducibility, consensus, and accuracy of biomedical search retrieval models, demonstrating high reproducibility and accuracy of Gemini models compared to PubMed and expert curation.

Contribution

The study presents a novel pipeline for assessing biomedical search models, providing a systematic way to evaluate reproducibility and accuracy of LLMs like Gemini in biomedical knowledge retrieval.

Findings

Gemini models show ~99% run-to-run reproducibility.

Gemini achieves ~87% accuracy in consensus evaluations.

LLMs are reliable tools for complex biomedical knowledge navigation.

Abstract

A scientific study begins with a central question, and search engines like PubMed are the first tools for retrieving knowledge and understanding the current state of the art. Large Language Models (LLMs) have been used in research, promising acceleration and deeper results. However, besides caution, they demand rigorous validation. Assessing complex biological relationships remains challenging for SQL-based tools and LLM models. Here, we introduce the Digital Pathway Curation (DPC) pipeline to evaluate the reproducibility and accuracy of the Gemini models against PubMed search and human expert curation. Using two omics experiments, we created a large dataset (Ensemble) based on determining pathway-disease associations. With the Ensemble dataset, we demonstrate that Gemini achieves high run-to-run reproducibility of approximately 99% and inter-model reproducibility of around 75%. Next, we calculate the crowdsourced consensus using a smaller dataset. The CSC allows us to calculate accuracies, and the Gemini multi-model consensus reached a significant accuracy of about 87%. Our findings demonstrate that LLMs are reproducible, reliable, and valuable tools for navigating complex biomedical knowledge.