What Don't You Understand? Using Large Language Models to Identify and Characterize Student Misconceptions About Challenging Topics

TL;DR

This paper introduces a scalable method combining performance metrics and large language models to identify and characterize student misconceptions in online biomedical courses, enabling targeted educational interventions.

Contribution

It presents a novel two-stage methodology using LLMs and performance data to identify challenging topics and misconceptions in online learning environments.

Findings

Identified key challenging topics across multiple courses.

LLMs provided high-quality characterization of misconceptions.

Faculty found the method useful and aligned with their observations.

Abstract

This study presents a systematic approach to identifying and characterizing student misconceptions in online learning environments through a novel combination of quantitative performance analysis and large language model (LLM) assessment. We analyzed data from 9 course periods across 5 online biomedical science courses, encompassing 3,802 medical student enrollments. Using data from 40-50 topic-focused quizzes per course, we developed a two-stage methodology. First, we identified challenging central topics using quiz-level performance metrics. Second, we employed LLMs to characterize the underlying misconceptions in these high-priority areas. By examining student performance on first attempts across primarily multiple-choice questions (MCQs), we identified consistently challenging topics that were also central to course objectives. We then leveraged recent advances in generative AI to analyze three distinct data sources in combination: quiz question content, student response patterns, and lecture transcripts. This approach revealed actionable insights about student misconceptions that were not apparent from performance data alone. The quality of the LLM-identified misconceptions was rated as excellent by subject matter experts. We also conducted teacher interviews to assess the perceived utility of our topic identification method. Faculty found that data-driven identification of challenging topics was valuable and corroborated their own classroom observations. This methodology provides a scalable approach to characterizing student difficulties in learning environments where quizzes are used. Our findings demonstrate the potential for targeted and potentially personalized interventions in future course iterations, with clear pathways for measuring intervention effectiveness through follow-up quiz performance.