How Many Papers Should You Review? A Research Synthesis of Systematic Literature Reviews in Software Engineering

TL;DR

This study analyzes 170 systematic literature reviews in software engineering to understand typical dataset sizes and review periods, providing benchmarks to guide researchers on when conducting an SLR is appropriate.

Contribution

It offers empirical insights into the characteristics of datasets in SE SLRs, aiding researchers in deciding when to undertake an SLR.

Findings

Median dataset size is 57 papers.

Median review period is 14 years.

Weak correlation between dataset size and review period.

Abstract

[Context] Systematic Literature Review (SLR) has been a major type of study published in Software Engineering (SE) venues for about two decades. However, there is a lack of understanding of whether an SLR is really needed in comparison to a more conventional literature review. Very often, SE researchers embark on an SLR with such doubts. We aspire to provide more understanding of when an SLR in SE should be conducted. [Objective] The first step of our investigation was focused on the dataset, i.e., the reviewed papers, in an SLR, which indicates the development of a research topic or area. The objective of this step is to provide a better understanding of the characteristics of the datasets of SLRs in SE. [Method] A research synthesis was conducted on a sample of 170 SLRs published in top-tier SE journals. We extracted and analysed the quantitative attributes of the datasets of these SLRs. [Results] The findings show that the median size of the datasets in our sample is 57 reviewed papers, and the median review period covered is 14 years. The number of reviewed papers and review period have a very weak and non-significant positive correlation. [Conclusions] The results of our study can be used by SE researchers as an indicator or benchmark to understand whether an SLR is conducted at a good time.