Better Assumptions, Stronger Conclusions: The Case for Ordinal Regression in HCI

TL;DR

This paper advocates for the use of cumulative link models in HCI to improve the analysis of ordinal data like Likert scales, highlighting limitations of current methods and providing practical guidance.

Contribution

It introduces and demonstrates the application of cumulative link (mixed) models for ordinal data analysis in HCI, promoting better statistical practices.

Findings

Current methods often rely on questionable assumptions.

Cumulative link models provide more appropriate analysis.

Practical examples using R are provided.

Abstract

Despite the widespread use of ordinal measures in HCI, such as Likert-items, there is little consensus among HCI researchers on the statistical methods used for analysing such data. Both parametric and non-parametric methods have been extensively used within the discipline, with limited reflection on their assumptions and appropriateness for such analyses. In this paper, we examine recent HCI works that report statistical analyses of ordinal measures. We highlight prevalent methods used, discuss their limitations and spotlight key assumptions and oversights that diminish the insights drawn from these methods. Finally, we champion and detail the use of cumulative link (mixed) models (CLM/CLMM) for analysing ordinal data. Further, we provide practical worked examples of applying CLM/CLMMs using R to published open-sourced datasets. This work contributes towards a better understanding of the statistical methods used to analyse ordinal data in HCI and helps to consolidate practices for future work.