A Collection of Systematic Reviews in Computer Science

TL;DR

This paper introduces SR4CS, a large-scale, openly available collection of computer science systematic reviews designed to facilitate reproducible research on retrieval and screening automation.

Contribution

It provides a comprehensive dataset with reviews, search queries, and metadata, along with baseline experiments comparing retrieval methods and zero-shot language models.

Findings

Baseline experiments reveal differences in precision, recall, and ranking among retrieval paradigms.

Naive zero-shot Boolean query generation has notable limitations.

The dataset supports reproducible evaluation of systematic review automation methods.

Abstract

Systematic reviews are the standard method for synthesizing scientific evidence, but their creation requires substantial manual effort, particularly during retrieval and screening. While recent work has explored automating these steps, evaluation resources remain largely confined to the biomedical domain, limiting reproducible experimentation in other domains. This paper introduces SR4CS, a large-scale collection of systematic reviews in computer science, designed to support reproducible research on Boolean query generation, retrieval, and screening. The corpus comprises 1,212 systematic reviews with their original expert-designed Boolean search queries, 104,316 resolved references, and structured methodological metadata. For controlled evaluation, the original Boolean queries are additionally provided in a normalized, approximated form operating over titles and abstracts. To illustrate the intended use of the collection, baseline experiments compare the approximated expert Boolean queries with zero-shot LLM-generated Boolean queries, BM25, and dense retrieval under a unified evaluation setting. The results highlight systematic differences in precision, recall, and ranking behavior across retrieval paradigms and expose limitations of naive zero-shot Boolean generation. SR4CS is released under an open license on Zenodo (https://doi.org/10.5281/zenodo.17163932), together with documentation and code (https://github.com/webis-de/scolia26-sr4cs), to enable reproducible evaluation and future research on scaling systematic review automation.