Algebra Error Classification with Large Language Models

TL;DR

This paper presents a novel approach using large language models for algebra error classification, overcoming limitations of rule-based and syntax-tree dependent methods, and improving classification accuracy and coverage.

Contribution

Introduces a flexible, LLM-based method for algebra error classification that outperforms existing approaches and handles unstructured student responses.

Findings

Outperforms existing error classification methods

Classifies a broader range of student responses

Identifies common classification errors and discusses limitations

Abstract

Automated feedback as students answer open-ended math questions has significant potential in improving learning outcomes at large scale. A key part of automated feedback systems is an error classification component, which identifies student errors and enables appropriate, predefined feedback to be deployed. Most existing approaches to error classification use a rule-based method, which has limited capacity to generalize. Existing data-driven methods avoid these limitations but specifically require mathematical expressions in student responses to be parsed into syntax trees. This requirement is itself a limitation, since student responses are not always syntactically valid and cannot be converted into trees. In this work, we introduce a flexible method for error classification using pre-trained large language models. We demonstrate that our method can outperform existing methods in algebra error classification, and is able to classify a larger set of student responses. Additionally, we analyze common classification errors made by our method and discuss limitations of automated error classification.