Retrieving and mining professional experience of software practice from grey literature: an exploratory review

TL;DR

This exploratory review finds a lack of research on mining practitioners' self-reports of software practice experience, highlighting significant challenges and opportunities for future investigation.

Contribution

It is the first review to classify existing work and identify key challenges in retrieving and mining professional experience reports in software engineering.

Findings

Only one relevant article found, focusing on emotional experiences rather than practice reports.

Major challenges include data scarcity, varied definitions, and evaluation difficulties.

No prior research directly addresses mining professional experience reports.

Abstract

Background: Retrieving and mining practitioners' self--reports of their professional experience of software practice could provide valuable evidence for research. We are, however, unaware of any existing reviews of research conducted in this area. Objective: To review and classify previous research, and to identify insights into the challenges research confronts when retrieving and mining practitioners' self-reports of their experience of software practice. Method: We conduct an exploratory review to identify and classify 42 articles. We analyse a selection of those articles for insights on challenges to mining professional experience. Results: We identify only one directly relevant article. Even then this article concerns the software professional's emotional experiences rather than the professional's reporting of behaviour and events occurring during software practice. We discuss challenges concerning: the prevalence of professional experience; definitions, models and theories; the sparseness of data; units of discourse analysis; annotator agreement; evaluation of the performance of algorithms; and the lack of replications. Conclusion: No directly relevant prior research appears to have been conducted in this area. We discuss the value of reporting negative results in secondary studies. There are a range of research opportunities but also considerable challenges. We formulate a set of guiding questions for further research in this area.