Comparing large language models for supervised analysis of students' lab notes

TL;DR

This study compares various large language models and traditional methods for analyzing students' lab notes, focusing on their performance, resource use, and research implications in physics education.

Contribution

It provides a comparative analysis of fine-tuned and few-shot LLMs versus traditional methods for classifying student lab notes in physics education research.

Findings

Higher-resource models often perform better but not always.

All models show similar research trend estimations.

Absolute measurement values vary beyond uncertainties.

Abstract

Recent advancements in large language models (LLMs) hold significant promise in improving physics education research that uses machine learning. In this study, we compare the application of various models to perform large-scale analysis of written text grounded in a physics education research classification problem: identifying skills in students' typed lab notes through sentence-level labeling. Specifically, we use training data to fine-tune two different LLMs, BERT and LLaMA, and compare the performance of these models to both a traditional bag of words approach and a few-shot LLM (without fine-tuning).} We evaluate the models based on their resource use, performance metrics, and research outcomes when identifying skills in lab notes. We find that higher-resource models often, but not necessarily, perform better than lower-resource models. We also find that all models estimate similar trends in research outcomes, although the absolute values of the estimated measurements are not always within uncertainties of each other. We use the results to discuss relevant considerations for education researchers seeking to select a model type to use as a classifier.