Automatic deductive coding in discourse analysis: an application of large language models in learning analytics

TL;DR

This paper evaluates the effectiveness of large language models, especially GPT with prompt engineering, in automating deductive coding in discourse analysis, showing superior performance over traditional methods.

Contribution

It introduces a novel application of GPT and prompt engineering for automatic deductive coding, demonstrating improved accuracy with limited training data.

Findings

GPT with prompt engineering outperforms traditional and BERT-like methods

Large language models require fewer training samples for accurate coding

Detailed prompt structures enhance GPT's coding performance

Abstract

Deductive coding is a common discourse analysis method widely used by learning science and learning analytics researchers for understanding teaching and learning interactions. It often requires researchers to manually label all discourses to be analyzed according to a theoretically guided coding scheme, which is time-consuming and labor-intensive. The emergence of large language models such as GPT has opened a new avenue for automatic deductive coding to overcome the limitations of traditional deductive coding. To evaluate the usefulness of large language models in automatic deductive coding, we employed three different classification methods driven by different artificial intelligence technologies, including the traditional text classification method with text feature engineering, BERT-like pretrained language model and GPT-like pretrained large language model (LLM). We applied these methods to two different datasets and explored the potential of GPT and prompt engineering in automatic deductive coding. By analyzing and comparing the accuracy and Kappa values of these three classification methods, we found that GPT with prompt engineering outperformed the other two methods on both datasets with limited number of training samples. By providing detailed prompt structures, the reported work demonstrated how large language models can be used in the implementation of automatic deductive coding.