Reliability in Software Engineering Qualitative Research through Inter-Coder Agreement: A guide using Krippendorff's $\alpha$ & Atlas.ti

TL;DR

This paper introduces a comprehensive framework for assessing the reliability of qualitative coding in software engineering research, emphasizing Krippendorff's alpha and providing practical guidance with Atlas.ti.

Contribution

It unifies various Krippendorff's alpha variants into a single mathematical framework and offers a detailed methodological guide for empirical researchers.

Findings

Unified Krippendorff's alpha variants into one framework

Provided calculation and interpretation examples using Atlas.ti

Enhanced reliability assessment in qualitative software engineering studies

Abstract

In recent years, the research on empirical software engineering that uses qualitative data analysis (e.g., cases studies, interview surveys, and grounded theory studies) is increasing. However, most of this research does not deep into the reliability and validity of findings, specifically in the reliability of coding in which these methodologies rely on, despite there exist a variety of statistical techniques known as Inter-Coder Agreement (ICA) for analyzing consensus in team coding. This paper aims to establish a novel theoretical framework that enables a methodological approach for conducting this validity analysis. This framework is based on a set of coefficients for measuring the degree of agreement that different coders achieve when judging a common matter. We analyze different reliability coefficients and provide detailed examples of calculation, with special attention to Krippendorff's $\alpha$ coefficients. We systematically review several variants of Krippendorff's $\alpha$ reported in the literature and provide a novel common mathematical framework in which all of them are unified through a universal $\alpha$ coefficient. Finally, this paper provides a detailed guide of the use of this theoretical framework in a large case study on DevOps culture. We explain how $\alpha$ coefficients are computed and interpreted using a widely used software tool for qualitative analysis like Atlas.ti. We expect that this work will help empirical researchers, particularly in software engineering, to improve the quality and trustworthiness of their studies.