Empowering Computing Education Researchers Through LLM-Assisted Content Analysis

TL;DR

This paper introduces a novel LLM-assisted content analysis method that enables computing education researchers to analyze large volumes of textual data more efficiently and rigorously, thereby enhancing research scope and generalizability.

Contribution

It proposes a new method, LACA, combining content analysis with large language models to facilitate large-scale, rigorous qualitative data analysis in computing education research.

Findings

Demonstrated LACA on a computing education dataset

Showed LACA enables analysis of larger datasets with less researcher burden

Highlighted potential for more generalizable CER findings

Abstract

Computing education research (CER) is often instigated by practitioners wanting to improve both their own and the wider discipline's teaching practice. However, the latter is often difficult as many researchers lack the colleagues, resources, or capacity to conduct research that is generalisable or rigorous enough to advance the discipline. As a result, research methods that enable sense-making with larger volumes of qualitative data, while not increasing the burden on the researcher, have significant potential within CER. In this discussion paper, we propose such a method for conducting rigorous analysis on large volumes of textual data, namely a variation of LLM-assisted content analysis (LACA). This method combines content analysis with the use of large language models, empowering researchers to conduct larger-scale research which they would otherwise not be able to perform. Using a computing education dataset, we illustrate how LACA could be applied in a reproducible and rigorous manner. We believe this method has potential in CER, enabling more generalisable findings from a wider range of research. This, together with the development of similar methods, can help to advance both the practice and research quality of the CER discipline.