When LLMs Help -- and Hurt -- Teaching Assistants in Proof-Based Courses
Romina Mahinpei, Sofiia Druchyna, Manoel Horta Ribeiro

TL;DR
This study investigates the potential and limitations of using Large Language Models to assist teaching assistants in grading and providing feedback in proof-based courses, highlighting both benefits and challenges.
Contribution
It provides a detailed case study comparing LLM and TA grading decisions and perceptions, informing future human-AI collaboration in education.
Findings
Substantial disagreement between LLMs and TAs on grading decisions.
LLM-generated feedback is helpful for submissions with major errors.
Insights into design implications for AI-assisted grading systems.
Abstract
Teaching assistants (TAs) are essential to grading and feedback provision in proof-based courses, yet these tasks are time-intensive and difficult to scale. Although Large Language Models (LLMs) have been studied for grading and feedback, their effectiveness in proof-based courses is still unknown. Before designing LLM-based systems for this context, a necessary prerequisite is to understand whether LLMs can meaningfully assist TAs with grading and feedback. As such, we present a multi-part case study functioning as a technology probe in an undergraduate proof-based course. We compare rubric-based grading decisions made by an LLM and TAs with varying levels of expertise and examine TAs' perceptions of feedback generated by an LLM. We find substantial disagreement between LLMs and TAs on grading decisions but that LLM-generated feedback can still be useful to TAs for submissions with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIntelligent Tutoring Systems and Adaptive Learning · Topic Modeling · Explainable Artificial Intelligence (XAI)
