Can LLMs Identify Gaps and Misconceptions in Students' Code Explanations?

TL;DR

This paper explores how different large language model techniques, including prompting, fine-tuning, and preference alignment, can effectively identify gaps and misconceptions in students' code explanations, with GPT-4 showing superior performance.

Contribution

The study compares multiple LLM approaches for identifying misconceptions in student explanations, highlighting the effectiveness of fine-tuning and preference optimization methods.

Findings

GPT-4 outperforms LLaMA3 and Mistral in gap detection.

Fine-tuned LLMs are more effective than zero-shot prompting.

Preference optimization (ORPO) surpasses supervised fine-tuning in accuracy.

Abstract

This paper investigates various approaches using Large Language Models (LLMs) to identify gaps and misconceptions in students' self-explanations of specific instructional material, in our case explanations of code examples. This research is a part of our larger effort to automate the assessment of students' freely generated responses, focusing specifically on their self-explanations of code examples during activities related to code comprehension. In this work, we experiment with zero-shot prompting, Supervised Fine-Tuning (SFT), and preference alignment of LLMs to identify gaps in students' self-explanation. With simple prompting, GPT-4 consistently outperformed LLaMA3 and Mistral in identifying gaps and misconceptions, as confirmed by human evaluations. Additionally, our results suggest that fine-tuned large language models are more effective at identifying gaps in students' explanations compared to zero-shot and few-shot prompting techniques. Furthermore, our findings show that the preference optimization approach using Odds Ratio Preference Optimization (ORPO) outperforms SFT in identifying gaps and misconceptions in students' code explanations.