Clustering Introductory Computer Science Exercises Using Topic Modeling Methods

TL;DR

This paper explores the use of topic modeling techniques to automatically cluster introductory computer science exercises, transforming code solutions into text and validating the semantic coherence of the resulting clusters.

Contribution

It introduces a novel method combining code structure analysis with topic modeling to identify meaningful question clusters in computer science education.

Findings

Six semantically coherent clusters identified

Achieved 0.75 NPMI score indicating strong semantic coherence

Results correlate with human expert ratings

Abstract

Manually determining concepts present in a group of questions is a challenging and time-consuming process. However, the process is an essential step while modeling a virtual learning environment since a mapping between concepts and questions using mastery level assessment and recommendation engines are required. We investigated unsupervised semantic models (known as topic modeling techniques) to assist computer science teachers in this task and propose a method to transform Computer Science 1 teacher-provided code solutions into representative text documents, including the code structure information. By applying non-negative matrix factorization and latent Dirichlet allocation techniques, we extract the underlying relationship between questions and validate the results using an external dataset. We consider the interpretability of the learned concepts using 14 university professors' data, and the results confirm six semantically coherent clusters using the current dataset. Moreover, the six topics comprise the main concepts present in the test dataset, achieving 0.75 in the normalized pointwise mutual information metric. The metric correlates with human ratings, making the proposed method useful and providing semantics for large amounts of unannotated code.