Practitioner-generated blog posts as evidence for software engineering research: attitudinal survey and preliminary checklist

TL;DR

This study investigates the use of practitioner-generated blog posts as evidence in software engineering research, highlighting challenges in assessing credibility and proposing preliminary guidelines to improve their utilization.

Contribution

It provides an analysis of researcher attitudes towards blog posts as evidence and introduces initial checklists to guide their evaluation in SE research.

Findings

No consensus on credibility assessment criteria.

Blog posts often reflect prior beliefs without citations.

Proposed checklists aim to improve evidence evaluation.

Abstract

Background: Blog posts are frequently used by software practitioners to share information about their practice. Blog posts therefore provide a potential source of evidence for software engineering (SE) research. The use of blog posts as evidence for research appears contentious amongst some SE researchers. Objective: To better understand the actual and perceived value of blog posts as evidence for SE research, and to develop guidance for SE researchers on the use of blog posts as evidence. Method: We further analyse responses from a previously conducted attitudinal survey of 44 software engineering researchers. We conduct a heatmap analysis, simple statistical analysis, and a thematic analysis. Results: We find no clear consensus from respondents on researchers' attitudes to the credibility of blog posts, or on a standard set of criteria to evaluate blog-post credibility. We show that some of the responses to the survey exhibit characteristics similar to the content of blog posts, e.g., asserting prior beliefs as claims, with no citations and little supporting rationale. We illustrate our insights with ~60 qualitative examples from the survey ~40% of the total responses. We complement our quantitative and qualitative analyses with preliminary checklists to guide SE researchers. Conclusion: Blog posts are relevant to research because they are written by software practitioners describing their practice and experience. But evaluating the credibility of blog posts, so as to select the higher-quality content, remains an ongoing challenge. The quantitative and qualitative results, with the proposed checklists, are intended to stimulate reflection and action in the research community on the role of blog posts as evidence in software engineering research. Finally, our findings on researchers' attitudes to blog posts also provide more general insights into researchers' values for SE research.