CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews

TL;DR

This paper introduces CSMeD, a comprehensive meta-dataset consolidating multiple datasets for automated citation screening in systematic literature reviews, addressing evaluation challenges and providing a resource for training and benchmarking models.

Contribution

The paper presents CSMeD, a unified meta-dataset for automated citation screening, and CSMeD-FT, a new dataset for full text screening, improving evaluation consistency and model training.

Findings

CSMeD consolidates nine datasets with 325 SLRs from medicine and computer science.

Baseline experiments demonstrate the utility of CSMeD for training and evaluating models.

CSMeD addresses issues of dataset size, data leakage, and applicability in automated literature screening.

Abstract

Systematic literature reviews (SLRs) play an essential role in summarising, synthesising and validating scientific evidence. In recent years, there has been a growing interest in using machine learning techniques to automate the identification of relevant studies for SLRs. However, the lack of standardised evaluation datasets makes comparing the performance of such automated literature screening systems difficult. In this paper, we analyse the citation screening evaluation datasets, revealing that many of the available datasets are either too small, suffer from data leakage or have limited applicability to systems treating automated literature screening as a classification task, as opposed to, for example, a retrieval or question-answering task. To address these challenges, we introduce CSMeD, a meta-dataset consolidating nine publicly released collections, providing unified access to 325 SLRs from the fields of medicine and computer science. CSMeD serves as a comprehensive resource for training and evaluating the performance of automated citation screening models. Additionally, we introduce CSMeD-FT, a new dataset designed explicitly for evaluating the full text publication screening task. To demonstrate the utility of CSMeD, we conduct experiments and establish baselines on new datasets.