Mining the Gold: Student-AI Chat Logs as Rich Sources for Automated Knowledge Gap Detection

TL;DR

This paper introduces QueryQuilt, a multi-agent LLM framework that analyzes student-AI chat logs to automatically detect and quantify knowledge gaps in large lectures, aiming to improve teaching effectiveness.

Contribution

It presents a novel multi-agent LLM system that identifies class-wide knowledge gaps from student-AI dialogues, enhancing real-time instructional insights.

Findings

100% accuracy in simulated student data

95% completeness on real student-AI dialogues

Potential to improve teaching in large classes

Abstract

With the significant increase in enrollment in computing-related programs over the past 20 years, lecture sizes have grown correspondingly. In large lectures, instructors face challenges on identifying students' knowledge gaps timely, which is critical for effective teaching. Existing classroom response systems rely on instructor-initiated interactions, which limits their ability to capture the spontaneous knowledge gaps that naturally emerge during lectures. With the widespread adoption of LLMs among students, we recognize these student-AI dialogues as a valuable, student-centered data source for identifying knowledge gaps. In this idea paper, we propose QueryQuilt, a multi-agent LLM framework that automatically detects common knowledge gaps in large-scale lectures by analyzing students' chat logs with AI assistants. QueryQuilt consists of two key components: (1) a Dialogue Agent that responds to student questions while employing probing questions to reveal underlying knowledge gaps, and (2) a Knowledge Gap Identification Agent that systematically analyzes these dialogues to identify knowledge gaps across the student population. By generating frequency distributions of identified gaps, instructors can gain comprehensive insights into class-wide understanding. Our evaluation demonstrates promising results, with QueryQuilt achieving 100% accuracy in identifying knowledge gaps among simulated students and 95% completeness when tested on real student-AI dialogue data. These initial findings indicate the system's potential for facilitate teaching in authentic learning environments. We plan to deploy QueryQuilt in actual classroom settings for comprehensive evaluation, measuring its detection accuracy and impact on instruction.