Interpreting Latent Student Knowledge Representations in Programming Assignments

TL;DR

This paper introduces InfoOIRT, a model that interprets latent student knowledge in programming assignments by generating code and learning disentangled, interpretable representations of student skills and code features.

Contribution

The paper presents a novel information regularized model that enhances interpretability of student knowledge representations while accurately generating student code.

Findings

InfoOIRT effectively generates student code.

It produces interpretable, disentangled knowledge representations.

The model performs well on real-world programming data.

Abstract

Recent advances in artificial intelligence for education leverage generative large language models, including using them to predict open-ended student responses rather than their correctness only. However, the black-box nature of these models limits the interpretability of the learned student knowledge representations. In this paper, we conduct a first exploration into interpreting latent student knowledge representations by presenting InfoOIRT, an Information regularized Open-ended Item Response Theory model, which encourages the latent student knowledge states to be interpretable while being able to generate student-written code for open-ended programming questions. InfoOIRT maximizes the mutual information between a fixed subset of latent knowledge states enforced with simple prior distributions and generated student code, which encourages the model to learn disentangled representations of salient syntactic and semantic code features including syntactic styles, mastery of programming skills, and code structures. Through experiments on a real-world programming education dataset, we show that InfoOIRT can both accurately generate student code and lead to interpretable student knowledge representations.