Who's actually being Studied? A Call for Population Analysis in Software Engineering Research

TL;DR

This paper emphasizes the importance of population analysis in empirical software engineering to improve research validity and calls for better population framing and analysis methods.

Contribution

It highlights the gap in population analysis in ESE and advocates for systematic approaches to improve external validity and generalizability.

Findings

Identifies challenges in analyzing populations of engineers, organizations, and projects.

Argues for the need of appropriate population frames in ESE research.

Calls for improved methods to enhance empirical rigor and external validity.

Abstract

Population analysis is crucial for ensuring that empirical software engineering (ESE) research is representative and its findings are valid. Yet, there is a persistent gap between sampling processes and the holistic examination of populations, which this position paper addresses. We explore the challenges ranging from analysing populations of individual software engineers to organizations and projects. We discuss the interplay between generalizability and transferability and advocate for appropriate population frames. We also present a compelling case for improved population analysis aiming to enhance the empirical rigor and external validity of ESE research.