Establishing a Search String to Detect Secondary Studies in Software Engineering

TL;DR

This paper proposes and validates an effective search string for detecting secondary studies in Software Engineering, achieving over 90% recall and nearly 60% precision, facilitating easier identification of related reviews.

Contribution

It introduces a validated search string for secondary studies in SE, addressing challenges in recall, relevance, and precision, based on analysis of multiple tertiary studies.

Findings

Over 90% recall in retrieving secondary studies

High general precision of nearly 60%

Effective search string includes specific review-related terms

Abstract

Context: A tertiary study can be performed to identify related reviews on a topic of interest. However, the elaboration of an appropriate and effective search string to detect secondary studies is challenging for Software Engineering (SE) researchers. Objective: The main goal of this study is to propose a suitable search string to detect secondary studies in SE, addressing issues such as the quantity of applied terms, relevance, recall and precision. Method: We analyzed seven tertiary studies under two perspectives: (1) structure -- strings' terms to detect secondary studies; and (2) field: where searching -- titles alone or abstracts alone or titles and abstracts together, among others. We validate our string by performing a two-step validation process. Firstly, we evaluated the capability to retrieve secondary studies over a set of 1537 secondary studies included in 24 tertiary studies in SE. Secondly, we evaluated the general capacity of retrieving secondary studies over an automated search using the Scopus digital library. Results: Our string was capable to retrieve an optimum value of over 90\% of the included secondary studies (recall) with a high general precision of almost 60\%. Conclusion: The suitable search string for finding secondary studies in SE contains the terms "systematic review", "literature review", "systematic mapping", "mapping study" and "systematic map".