Adoption and implication of the Biased-Annotator Competence Estimation (BACE) model into COVID-19 vaccine Twitter data: Human annotation for latent message features

TL;DR

This paper demonstrates how the BACE Bayesian model improves human coding accuracy and accounts for coder biases in analyzing COVID-19 vaccine Twitter data, offering a more nuanced understanding of latent message features.

Contribution

It applies the BACE model to real-world Twitter data, comparing it with traditional methods, and discusses its potential to enhance human annotation of latent message features.

Findings

BACE accounts for coder biases and reliability.

Compared BACE with other statistical models.

Improves estimation of latent message features.

Abstract

Traditional quantitative content analysis approach (human coding method) has weaknesses, such as assuming all human coders are equally accurate once the intercoder reliability for training reaches a threshold score. We applied the Biased-Annotator Competence Estimation (BACE) model (Tyler, 2021), which draws on Bayesian modeling to improve human coding. An important contribution of this model is it takes each coder's potential biases and reliability into consideration and treats the "true" label of each message as a latent parameter, with quantifiable estimation uncertainties. In contrast, in conventional human coding, each message will receive a fixed label without estimates for measurement uncertainties. In this extended abstract, we first summarize the weaknesses of conventional human coding; and then apply the BACE model to COVID-19 vaccine Twitter data and compare BACE with other statistical models; finally, we discuss how the BACE model can be applied to improve human coding of latent message features.