Option Tracing: Beyond Correctness Analysis in Knowledge Tracing

TL;DR

This paper extends knowledge tracing methods to predict specific student choices in multiple choice questions, enabling detailed error diagnosis beyond overall correctness, and demonstrates their effectiveness on large datasets.

Contribution

It introduces option tracing, a novel extension of knowledge tracing that predicts specific student responses, facilitating detailed error analysis.

Findings

Effective in predicting student options in large datasets

Able to identify clusters of common student errors

Improves diagnostic capabilities over traditional correctness-based methods

Abstract

Knowledge tracing refers to a family of methods that estimate each student's knowledge component/skill mastery level from their past responses to questions. One key limitation of most existing knowledge tracing methods is that they can only estimate an \emph{overall} knowledge level of a student per knowledge component/skill since they analyze only the (usually binary-valued) correctness of student responses. Therefore, it is hard to use them to diagnose specific student errors. In this paper, we extend existing knowledge tracing methods beyond correctness prediction to the task of predicting the exact option students select in multiple choice questions. We quantitatively evaluate the performance of our option tracing methods on two large-scale student response datasets. We also qualitatively evaluate their ability in identifying common student errors in the form of clusters of incorrect options across different questions that correspond to the same error.