ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research

TL;DR

This paper introduces ScIRGen, a large-scale, realistic scientific QA and retrieval dataset created using novel data augmentation and question generation techniques, to better reflect researchers' real information needs.

Contribution

We developed a new dataset generation framework, ScIRGen, that produces realistic scientific questions and answers, filling the gap in existing datasets and enabling better evaluation of scientific retrieval and QA methods.

Findings

Current methods struggle with reasoning on complex scientific questions.

The ScIRGen-Geo dataset contains 61,000 QA pairs.

Benchmarking shows room for improvement in scientific QA and retrieval.

Abstract

Scientific researchers need intensive information about datasets to effectively evaluate and develop theories and methodologies. The information needs regarding datasets are implicitly embedded in particular research tasks, rather than explicitly expressed in search queries. However, existing scientific retrieval and question-answering (QA) datasets typically address straightforward questions, which do not align with the distribution of real-world research inquiries. To bridge this gap, we developed ScIRGen, a dataset generation framework for scientific QA \& retrieval that more accurately reflects the information needs of professional science researchers, and uses it to create a large-scale scientific retrieval-augmented generation (RAG) dataset with realistic queries, datasets and papers. Technically, we designed a dataset-oriented information extraction method that leverages academic papers to augment the dataset representation. We then proposed a question generation framework by employing cognitive taxonomy to ensure the quality of synthesized questions. We also design a method to automatically filter synthetic answers based on the perplexity shift of LLMs, which is highly aligned with human judgment of answers' validity. Collectively, these methodologies culminated in the creation of the 61k QA dataset, ScIRGen-Geo. We benchmarked representative methods on the ScIRGen-Geo dataset for their question-answering and retrieval capabilities, finding out that current methods still suffer from reasoning from complex questions. This work advances the development of more sophisticated tools to support the intricate information needs of the scientific community.