Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning system

TL;DR

This study develops a framework integrating large language models with knowledge graphs for programming education, demonstrating that hybrid AI modes improve learner performance and are well-received.

Contribution

It introduces a novel hybrid GenAI-adaptive feedback system for programming learning, combining knowledge graphs and user data for enhanced feedback and recommendations.

Findings

Hybrid GenAI-adaptive mode yields highest correct submissions.

GenAI feedback is perceived as highly helpful.

All modes are rated positively for usability.

Abstract

This paper introduces the design and development of a framework that integrates a large language model (LLM) with a retrieval-augmented generation (RAG) approach leveraging both a knowledge graph and user interaction history. The framework is incorporated into a previously developed adaptive learning support system to assess learners' code, generate formative feedback, and recommend exercises. Moerover, this study examines learner preferences across three instructional modes; adaptive, Generative AI (GenAI), and hybrid GenAI-adaptive. An experimental study was conducted to compare the learning performance and perception of the learners, and the effectiveness of these three modes using four key log features derived from 4956 code submissions across all experimental groups. The analysis results show that learners receiving feedback from GenAI modes had significantly more correct code and fewer code submissions missing essential programming logic than those receiving feedback from adaptive mode. In particular, the hybrid GenAI-adaptive mode achieved the highest number of correct submissions and the fewest incorrect or incomplete attempts, outperforming both the adaptive-only and GenAI-only modes. Questionnaire responses further indicated that GenAI-generated feedback was widely perceived as helpful, while all modes were rated positively for ease of use and usefulness. These results suggest that the hybrid GenAI-adaptive mode outperforms the other two modes across all measured log features.