VILLA: Versatile Information Retrieval From Scientific Literature Using Large LAnguage Models

TL;DR

This paper introduces VILLA, a new retrieval-augmented generation framework for extracting complex mutation information from scientific literature in virology, addressing a gap in open-ended, domain-specific scientific information extraction.

Contribution

The paper presents VILLA, a novel multi-step RAG framework tailored for virology, along with a curated dataset of influenza mutations, advancing open-ended scientific information extraction.

Findings

VILLA outperforms baseline RAG and other state-of-the-art tools.

Curated dataset of 629 influenza mutations from 239 publications.

Demonstrates effectiveness in complex, open-ended SIE tasks.

Abstract

The lack of high-quality ground truth datasets to train machine learning (ML) models impedes the potential of artificial intelligence (AI) for science research. Scientific information extraction (SIE) from the literature using LLMs is emerging as a powerful approach to automate the creation of these datasets. However, existing LLM-based approaches and benchmarking studies for SIE focus on broad topics such as biomedicine and chemistry, are limited to choice-based tasks, and focus on extracting information from short and well-formatted text. The potential of SIE methods in complex, open-ended tasks is considerably under-explored. In this study, we used a domain that has been virtually ignored in SIE, namely virology, to address these research gaps. We design a unique, open-ended SIE task of extracting mutations in a given virus that modify its interaction with the host. We develop a new, multi-step retrieval augmented generation (RAG) framework called VILLA for SIE. In parallel, we curate a novel dataset of 629 mutations in ten influenza A virus proteins obtained from 239 scientific publications to serve as ground truth for the mutation extraction task. Finally, we demonstrate VILLA's superior performance using a novel and comprehensive evaluation and comparison with vanilla RAG and other state-of-the art RAG- and agent-based tools for SIE.