Modeling Programming Skills with Source Code Embeddings for Context-aware Exercise Recommendation

TL;DR

This paper introduces a context-aware recommender system that models students' programming skills using source code embeddings to improve exercise recommendations in an educational setting.

Contribution

It presents a novel skill modeling approach using source code embeddings and demonstrates its effectiveness over existing methods for personalized exercise recommendation.

Findings

Jina embeddings outperform other embedding methods in skill prediction.

The system provides more suitable exercise recommendations than baselines.

Source code embeddings effectively capture programming skills for recommendation.

Abstract

In this paper, we propose a context-aware recommender system that models students' programming skills using embeddings of the source code they submit throughout a course. These embeddings predict students' skills across multiple programming topics, producing profiles that are matched to the skills required by unseen homework problems. To generate recommendations, we compute the cosine similarity between student profiles and problem skill vectors, ranking exercises according to their alignment with each student's current abilities. We evaluated our approach using real data from students and exercises in an introductory programming course at our university. First, we assessed the effectiveness of our source code embeddings for predicting skills, comparing them with token-based and graph-based alternatives. Results showed that Jina embeddings outperformed TF-IDF, CodeBERT-cpp, and GraphCodeBERT across most skills. Additionally, we evaluated the system's ability to recommend exercises aligned with weekly course content by analyzing student submissions collected over seven course offerings. Our approach consistently produced more suitable recommendations than baselines based on correctness or solution time, indicating that predicted programming skills provide a stronger signal for problem recommendation.