Misclassification in Automated Content Analysis Causes Bias in Regression. Can We Fix It? Yes We Can!

TL;DR

Automated classifiers in communication research often suffer from misclassification bias, but with proper error correction methods, including a new R package, accurate and reliable analysis is achievable.

Contribution

The paper introduces a new error correction method and an R package to address misclassification bias in automated content analysis.

Findings

The new method effectively corrects bias in simulated data.

Automated classifiers can be useful with proper error correction.

Misclassification bias is often overlooked in communication science.

Abstract

Automated classifiers (ACs), often built via supervised machine learning (SML), can categorize large, statistically powerful samples of data ranging from text to images and video, and have become widely popular measurement devices in communication science and related fields. Despite this popularity, even highly accurate classifiers make errors that cause misclassification bias and misleading results in downstream analyses-unless such analyses account for these errors. As we show in a systematic literature review of SML applications, communication scholars largely ignore misclassification bias. In principle, existing statistical methods can use "gold standard" validation data, such as that created by human annotators, to correct misclassification bias and produce consistent estimates. We introduce and test such methods, including a new method we design and implement in the R package misclassificationmodels, via Monte Carlo simulations designed to reveal each method's limitations, which we also release. Based on our results, we recommend our new error correction method as it is versatile and efficient. In sum, automated classifiers, even those below common accuracy standards or making systematic misclassifications, can be useful for measurement with careful study design and appropriate error correction methods.