An Empirically Evaluated Checklist for Surveys in Software Engineering

TL;DR

This paper presents a systematically developed checklist for designing and assessing survey research in software engineering, aiming to improve research quality and provide a practical auditing tool.

Contribution

It introduces a new, empirically grounded checklist for survey research in software engineering, validated through expert evaluation and designed to support both design and assessment.

Findings

19 of 38 checklist items were improved after evaluation

The checklist helps audit survey reports and guide survey design

Evaluation revealed limitations in understanding and objectivity

Abstract

Context: Over the past decade Software Engineering research has seen a steady increase in survey-based studies, and there are several guidelines providing support for those willing to carry out surveys. The need for auditing survey research has been raised in the literature. Checklists have been used to assess different types of empirical studies, such as experiments and case studies. Objective: This paper proposes a checklist to support the design and assessment of survey-based research in software engineering grounded in existing guidelines for survey research. We further evaluated the checklist in the research practice context. Method: To construct the checklist, we systematically aggregated knowledge from 14 methodological papers supporting survey-based research in software engineering. We identified the key stages of the survey process and its recommended practices through thematic analysis and vote counting. To improve our initially designed checklist we evaluated it using a mixed evaluation approach involving experienced researchers. Results: The evaluation provided insights regarding limitations of the checklist in relation to its understanding and objectivity. In particular, 19 of the 38 checklist items were improved according to the feedback received from its evaluation. Finally, a discussion on how to use the checklist and what its implications are for research practice is also provided. Conclusion: The proposed checklist is an instrument suitable for auditing survey reports as well as a support tool to guide ongoing research with regard to the survey design process.