Case Survey Studies in Software Engineering Research

TL;DR

This paper reviews how the case survey method is used in Software Engineering research, highlighting inconsistencies and the need for clear guidelines to improve study quality and comparability.

Contribution

It provides a systematic mapping of case survey applications in SE and emphasizes the necessity for standardized guidelines for its proper use.

Findings

Heterogeneous understanding of case survey in SE studies

Inconsistent application of the method compared to seminal definitions

Need for clear guidelines to improve research quality

Abstract

Background: Given the social aspects of Software Engineering (SE), in the last twenty years, researchers from the field started using research methods common in social sciences such as case study, ethnography, and grounded theory. More recently, case survey, another imported research method, has seen its increasing use in SE studies. It is based on existing case studies reported in the literature and intends to harness the generalizability of survey and the depth of case study. However, little is known on how case survey has been applied in SE research, let alone guidelines on how to employ it properly. Aims: This article aims to provide a better understanding of how case survey has been applied in Software Engineering research. Method: To address this knowledge gap, we performed a systematic mapping study and analyzed 12 Software Engineering studies that used the case survey method. Results: Our findings show that these studies presented a heterogeneous understanding of the approach ranging from secondary studies to primary inquiries focused on a large number of instances of a research phenomenon. They have not applied the case survey method consistently as defined in the seminal methodological papers. Conclusions: We conclude that a set of clearly defined guidelines are needed on how to use case survey in SE research, to ensure the quality of the studies employing this approach and to provide a set of clearly defined criteria to evaluate such work.