RAGPPI: RAG Benchmark for Protein-Protein Interactions in Drug Discovery

TL;DR

This paper introduces RAGPPI, a new benchmark dataset of over 4,400 question-answer pairs focused on the biological impacts of protein-protein interactions, to support RAG systems in drug discovery.

Contribution

The paper presents the first benchmark dataset for PPI impact identification, including expert-annotated and auto-evaluated QA pairs, advancing RAG applications in drug discovery.

Findings

Created a gold-standard dataset of 500 QA pairs.

Developed an ensemble auto-evaluation LLM for dataset construction.

Established RAGPPI as a resource for the research community.

Abstract

Retrieving the biological impacts of protein-protein interactions (PPIs) is essential for target identification (Target ID) in drug development. Given the vast number of proteins involved, this process remains time-consuming and challenging. Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) frameworks have supported Target ID; however, no benchmark currently exists for identifying the biological impacts of PPIs. To bridge this gap, we introduce the RAG Benchmark for PPIs (RAGPPI), a factual question-answer benchmark of 4,420 question-answer pairs that focus on the potential biological impacts of PPIs. Through interviews with experts, we identified criteria for a benchmark dataset, such as a type of QA and source. We built a gold-standard dataset (500 QA pairs) through expert-driven data annotation. We developed an ensemble auto-evaluation LLM that reflected expert labeling characteristics, which facilitates the construction of a silver-standard dataset (3,720 QA pairs). We are committed to maintaining RAGPPI as a resource to support the research community in advancing RAG systems for drug discovery QA solutions.