A method to assess trustworthiness of machine coding at scale

TL;DR

This paper introduces a four-part method for assessing the trustworthiness of machine coding in physics education research, enabling reliable automated analysis of open-ended survey responses without full human cross-checking.

Contribution

It presents a novel framework combining model evaluation and uncertainty quantification to ensure trustworthy machine coding in natural language processing applications.

Findings

Validated the method with two different survey questions and coding schemes.

Demonstrated the approach on an unseen dataset to ensure reliability.

Provided practical recommendations for researchers using machine coding.

Abstract

Physics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more efficient and consistent than human coding, allowing much larger and broader data sets to be analyzed than is practical with human coders. Existing work that uses these tools, however, does not investigate norms that allow for trustworthy quantitative claims without full reliance on cross-checking with human coding, which defeats the purpose of using these automated tools. Here we propose a four-part method for making such claims with supervised natural language processing: evaluating a trained model, calculating statistical uncertainty, calculating systematic uncertainty from the trained algorithm, and calculating systematic uncertainty from novel data sources. We provide evidence for this method using data from two distinct short response survey questions with two distinct coding schemes. We also provide a real-world example of using these practices to machine code a data set unseen by human coders. We offer recommendations to guide physics education researchers who may use machine-coding methods in the future.