SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews

TL;DR

This paper introduces SESR-Eval, a benchmark dataset with over 34,000 labeled studies, to evaluate LLMs in software engineering systematic review screening, revealing current limitations in accuracy and cost-effectiveness.

Contribution

The paper provides the SESR-Eval dataset and benchmarking results, enabling assessment of LLM performance in SR screening within software engineering.

Findings

Most LLMs perform similarly in screening accuracy.

Differences between secondary studies exceed differences between LLMs.

Using an LLM costs less than $40 per secondary study.

Abstract

Background: The use of large language models (LLMs) in the title-abstract screening process of systematic reviews (SRs) has shown promising results, but suffers from limited performance evaluation. Aims: Create a benchmark dataset to evaluate the performance of LLMs in the title-abstract screening process of SRs. Provide evidence whether using LLMs in title-abstract screening in software engineering is advisable. Method: We start with 169 SR research artifacts and find 24 of those to be suitable for inclusion in the dataset. Using the dataset we benchmark title-abstract screening using 9 LLMs. Results: We present the SESR-Eval (Software Engineering Systematic Review Evaluation) dataset containing 34,528 labeled primary studies, sourced from 24 secondary studies published in software engineering (SE) journals. Most LLMs performed similarly and the differences in screening accuracy between secondary studies are greater than differences between LLMs. The cost of using an LLM is relatively low - less than $40 per secondary study even for the most expensive model. Conclusions: Our benchmark enables monitoring AI performance in the screening task of SRs in software engineering. At present, LLMs are not yet recommended for automating the title-abstract screening process, since accuracy varies widely across secondary studies, and no LLM managed a high recall with reasonable precision. In future, we plan to investigate factors that influence LLM screening performance between studies.