Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression

TL;DR

This paper extends the ORES model to produce continuous, one-dimensional quality scores for Wikipedia articles using weighted ordinal regression, addressing limitations of previous discrete, evenly spaced assumptions.

Contribution

It introduces a novel ordinal regression approach that improves accuracy and realism of Wikipedia article quality measurement over prior models.

Findings

Scores are correlated but more accurate for research datasets.

The 'evenly spaced' assumption is unsupported in practice.

Provides open code, data, and models for reproducibility.

Abstract

Organizing complex peer production projects and advancing scientific knowledge of open collaboration each depend on the ability to measure quality. Article quality ratings on English language Wikipedia have been widely used by both Wikipedia community members and academic researchers for purposes like tracking knowledge gaps and studying how political polarization shapes collaboration. Even so, measuring quality presents many methodological challenges. The most widely used systems use labels on discrete ordinal scales when assessing quality, but such labels can be inconvenient for statistics and machine learning. Prior work handles this by assuming that different levels of quality are "evenly spaced" from one another. This assumption runs counter to intuitions about the relative degrees of effort needed to raise Wikipedia encyclopedia articles to different quality levels. Furthermore, models from prior work are fit to datasets that oversample high-quality articles. This limits their accuracy for representative samples of articles or revisions. I describe a technique extending the Wikimedia Foundations' ORES article quality model to address these limitations. My method uses weighted ordinal regression models to construct one-dimensional continuous measures of quality. While scores from my technique and from prior approaches are correlated, my approach improves accuracy for research datasets and provides evidence that the "evenly spaced" assumption is unfounded in practice on English Wikipedia. I conclude with recommendations for using quality scores in future research and include the full code, data, and models.