Exploring Research Interest in Stack Overflow -- A Systematic Mapping Study and Quality Evaluation

TL;DR

This systematic mapping study analyzes the academic research on Stack Overflow, revealing increasing interest, common topics, types of contributions, and highlighting the need for validation of proposed solutions.

Contribution

It provides a comprehensive overview of the scale, focus, and quality of research on Stack Overflow, identifying gaps and areas for further validation.

Findings

Research interest in Stack Overflow has increased over the years.

Most studies are evaluative or propose solutions, but lack validation.

Only 58% of quality criteria are met in studies focused on software programming.

Abstract

Platforms such as Stack Overflow are available for software practitioners to solicit solutions to their challenges and knowledge needs. The practices therein have in recent times however triggered quality related concerns. This is a noteworthy issue when considering that the Stack Overflow platform is used by numerous software developers. Academic research tends to provide validation for the practices and processes employed by Stack Overflow and other such forums. However, previous work did not review the scale of scientific attention that is given to this cause. Continuing from our preliminary work, we conducted a Systematic Mapping study involving 265 papers from six relevant databases to address this gap. In this work, we explored the level of academic interest Stack Overflow has generated, the publication venues that are targeted, the topics that are studied, approaches used, types of contributions and the quality of the publications that are written about Stack Overflow. Outcomes show that Stack Overflow has attracted increasing research interest over the years, with topics relating to both community dynamics and human factors, and technical issues. In addition, research studies have been largely evaluative or proposed solutions; however, the latter approach tends to lack validation. The contributions of these studies are often techniques or answers to a specific problem. Evaluating the quality of all studies that were dedicated to software programming (58 papers), our outcomes show that on average only 58% of the developed quality criteria were met. Notwithstanding that research is continually aiming to understand Stack Overflow and other similar communities, further investigations are required to validate such studies and the solutions they propose.